Develop Reference

How to Account and Drop a Specific String before another string in Python

So I have an application that generally does a good job at collecting information and moving stuff from one AWS s3 bucket to another, and then processing it, but it doesn't really do a good job when people name their file with a pretext string.
Currently, I look for glob:
dump_files = glob.glob('docker-support*.zip')
What this does is I have logic built to only look for things that account for file names that utilize docker-support as the main identifier.
However, I need it to account for times when people do something like
super_Secret123-Production-whatever-docker-support*.zip
Basically, I would like for the function to rename it using that variable dump_files
Should I just set the variable to something like this:
dump_files = glob.glob('*docker-support*.zip')
or
dump_files = glob.glob('/^(.*?)\docker-support*.zip')
The main thing is I am going to want to pick it up, rename it and then strip the part of the file name that is before the actual file name needed for processing: docker-support*.zip as the application needs to look for files in S3 just named in that format.
Code that handles this:
#!/usr/bin/env python3
# main execution loop for dump analysis tool
# Author: Bryce Ryan, Mirantis Inc.
#
# checks for new files in dump_originals, when found, runs run-parts against that file
# v1.1
# pause.main.loop check
# improved error handling
# escape file name to run-parts to avoid metacharacters
#
#
import os
import tempfile
import time
import zipfile
import logging
import shutil
import glob
from datetime import date
import sys
from os import path
logging.basicConfig(filename='/dump/logs/analyzer_logs.txt', level=logging.DEBUG, format='%(asctime)s %(message)s', datefmt='%Y-%m-%dT%H:%M:%S%z' )
ROOT_DIR = os.path.abspath('..')
logging.debug("ROOT_DIR: {}".format(ROOT_DIR))
DUMP_DIR = os.path.join(ROOT_DIR, 'dump_originals')
logging.debug("DUMP_DIR: {}".format(DUMP_DIR))
WORK_DIR = os.path.join(ROOT_DIR, 'work_dir')
logging.debug("WORK_DIR: {}".format(WORK_DIR))
# can we actually create a file? just because we have perms, or think we do, doesn't mean there are
# enough inodes or capacity to do basic stuff.
with open(os.path.join(DUMP_DIR, "testfile"), 'w'):
pass
logging.info("Beginning event loop for lodestone. Looking for new files in {}".format(DUMP_DIR))
print("Beginning event loop for lodestone.")
sys.stdout.flush()
os.chdir(DUMP_DIR)
logging.basicConfig(filename="analyzer.logs", level=logging.DEBUG)
while True:
# here at the top of the loop, check to see if we should wait for a bit
# typically, because of testing or maintenance
# if the magic, undocumented file exists, wait for 5 sec and check again
# do this forever
while path.exists("/dump/pause.main.loop"):
print("Pausing main loop for 60s, waiting on /dump/pause.main.loop")
time.sleep(60)
dump_files = glob.glob('docker-support*.zip')
try:
if dump_files[0] != '':
logging.debug("files found")
print("================== BEGIN PROCESSING NEW FILE ========= ")
print("File found:", dump_files[0])
print(" ")
logging.info("Processing new file: " + dump_files[0] )
sys.stdout.flush()
support_dump_file = dump_files[0]
# check that it's an actual zip; if not, ignore it
if not zipfile.is_zipfile(support_dump_file):
print("File: " + str(support_dump_file))
print("Inbound file is not recognized as a zip file.\n\n")
logging.info("Inbound file not recognized as a zip file.")
# now move it out of the way so we don't see it again;
# ok if exists on destination and we ignore the error
shutil.move( support_dump_file, "../dump_complete/" )
# no further processing, so back to the top
sys.stdout.flush()
continue
temp_dir = tempfile.mkdtemp(prefix='dump.', dir=WORK_DIR)
os.chmod(temp_dir, 0o777)
logging.info("temp_dir is: " + temp_dir)
# cmd = ROOT_DIR + "/utilities/run-parts --exit-on-error --arg=analyze --arg=" + DUMP_DIR + " --arg=" + support_dump_file + " --arg=" + temp_dir + " " + ROOT_DIR + "/analysis"
cmd = ROOT_DIR + "/utilities/run-parts --arg=analyze --arg=" + DUMP_DIR + " --arg=\'" + support_dump_file + "\' --arg=" + temp_dir + " " + ROOT_DIR + "/analysis"
print(cmd)
logging.info("Will execute: " + cmd )
sys.stdout.flush()
try:
retcode = os.system(cmd)
tempdir =temp_dir
if retcode == 1:
print("Removing temporary work_dir")
logging.debug("Removing temporary work_dir", tempdir)
shutil.rmtree(tempdir, ignore_errors=True)
sys.stdout.flush()
finally:
print("Finally block for cmd. . .")
print("Removing temporary work_dir")
logging.debug("Removing work_dir " + tempdir)
print(tempdir)
sys.stdout.flush()
# shutil.rmtree(tempdir, ignore_errors=True)
os.system('/bin/rm -rf' + tempdir)
sys.stdout.flush()
except:
pass
# pause for a moment; save some processor cycles
sys.stdout.flush()
time.sleep(1)
Right now I do not have the function that will rename this in there.

FileNotFoundError even though Absolute Path Specified

I am running a simple python webserver [SimpleHTTPServer] in my Linux . Wrote a python program to download all the files hosted in that server to my Windows Machine . But for some reason program is throwing FileNotFoundError even though Directory exists and I've provided Absolute Path .
Here is the code : https://drive.google.com/file/d/1CDrueDJcbu2z1XeeB_iYv0zmfIX1cCkx/view?usp=sharing
It's working correctly in Linux but trouble with Windows . Thanks
import requests
import argparse
from sys import argv
from urllib.parse import unquote
import os
from time import time
import random
from colorama import Fore, Style
import platform
def formatFiles(name):
name = name[13:-9]
nameLen = len(name) - 2
nameLen = int(nameLen/2)
name = name[:nameLen]
return name
# Creating a Temporary Folder to Download all the files in it
def fileCreate(saveFolder):
random.seed(int(time()))
text = ""
for x in range(5):
y = random.randrange(65,91)
text += chr(y)
saveFolder += text
os.popen("mkdir {}".format(saveFolder))
print("Temp Directory {} created to save files/folders".format(text))
return saveFolder
def winDows(endPoint, banner):
resp = requests.get(endPoint, allow_redirects=True)
resp = resp.text.split("\n")
resp = list(map(unquote, resp[10:-5])) #URL decoding using unquote
resp = list(map(formatFiles,resp))
for dir in resp:
tempPath = ""
tempEndpoint = endPoint[len(serverURL):] # Getting directory structure by removing IP:PORT in URL
tempPath = "\\".join(tempEndpoint.split("/")) # Removing / and adding \\ for Windows path
print(banner + dir)
tdir = dir
if(dir[-1] == "/"):
if( dir.split(" ")!=1 ): # If the directory name has spaces ,
tdir = dir[:-1]
tdir = "\""+tdir+"\""+"\\"
os.popen("mkdir "+saveFolder+"\\"+tempPath+tdir)
r = winDows(endPoint+dir, banner[:-4]+" |___")
else:
data = open(saveFolder+"\\"+tempPath+dir, "wb")
fileData = requests.get(endPoint+dir, allow_redirects=True)
data.write(fileData.content)
data.close()
return 1
parser = argparse.ArgumentParser()
sideBanner = " |___ "
parser.add_argument("ip", help = "IP address of FTP Server", type=ip_valid)
parser.add_argument("port" , help = "FTP Server Port you want to access", type=port_valid)
parser.add_argument("dst", help="Destination Path to save your files")
args = parser.parse_args()
ip = argv[1]
port = argv[2]
saveFolder = argv[3]
serverURL = "http://"+ip+":"+port+"/"
saveFolder = fileCreate(saveFolder)
print("Destination Folder - {}".format(saveFolder))
if(platform.system() == "Linux"):
linuX(serverURL, sideBanner)
else:
winDows(serverURL, sideBanner)

python click set allowable values for option

I have created a Click command that will copy files from source to destination
The command accepts 3 parameters :
1 - Source of files
2 - Destination of files
3 - Transfer mode (local,ftp)
import click
#click.group()
def cli():
pass
#cli.command()
#click.argument('source')
#click.argument('destination')
#click.option('--mode', required = True)
def copy(source, destination, mode):
print("copying files from " + source + " to " + destination + "using " + mode + " mode")
if __name__ == '__main__':
cli()
When I call the script using this : command.py copy "C:/" "D:/" --mode=network
I get the following output : copying files from C:/ to D:/using network mode
As you can see I specified network as mode, but I want only two options : local or ftp
So how can I use Click to set the allowable values of an option ?

You want to use click.Choice
#cli.command()
#click.argument('source')
#click.argument('destination')
#click.option('--mode', type=click.Choice(['local', 'ftp']), required = True)
def copy(source, destination, mode):
print("copying files from " + source + " to " + destination + "using " + mode + " mode")

File Not Found Error Django

Hello I'm new to django and I'm trying to make a web app. I have a running back end, but the problem is it's only running on cli and I have to turn it into a web app.
def testing(request):
ksize = 6
somsize= 10
csvname="input.csv"
testcap = "testing.pcap"
pl.csv5("chap/a",testcap)
tmparr=[]
for filename in os.listdir("chap"):
if filename.endswith(".csv"):
tmparr.append(filename)
continue
else:
continue
tmparr.sort()
visual_list = natsort.natsorted(tmparr)
csv = sl.opencsv(csvname)
norm = sl.normalize(csv)
weights = sl.som(norm,somsize)
label = sl.kmeans(ksize,weights)
#for x in range (2,21):
# label = sl.kmeans(x,weights)
# print("K is", x, "Score is ", label[1])
lblarr = np.reshape(label,(somsize,somsize))
#sl.dispcolor(lblarr)
classess = sl.cluster_coloring(weights,norm,csv)
classpercluster = sl.determine_cluster(classess,lblarr,ksize)
classpercent = sl.toperc(classpercluster)
print (classpercent)
#print(classpercluster)
for x in visual_list:
temp = ("chap/"+x)
tests = sl.opencsv(temp)
print(tests)
hits = sl.som_hits(weights, tests)
name = ("img/" + x + ".png")
sl.disp(lblarr,name,hits)
return render(request,'visualization/detail.html')
The system cannot find the path specified: 'chap', I'm not sure if I should put the chap folder inside the templates folder or in the app folder. Thank you in advance!

You're doing relative paths here it looks like. Change it to an absolute path.
dirpath = os.path.dirname(os.path.abspath(__file__))
chap_dirpath = os.path.join(dirpath, chap_dirpath)

Python subprocess doesn't run until calling process finished

EDIT 1 - added more code
I'm not sure that proc.communicate was needed, it was one of the suggestions I found from some other stackoverflow code.(Sorry I was tired last night and didn't think too much before asking the question.)
I should add that I am not an experienced coder (mechanical engineer) as you can probably tell from my code
In my Gui I have a button to call a subprocess
The subprocess (screenshot-cmd.exe) creates a png of a cropped screen shot but it won't actually produce the file until there is an error or if the button click event is over.
This makes me think that the subprocess is not actually run until the event is finished
I want to call the process several times after a single button press and move the files that it produces after each one is produced
if I use proc.wait(), the process hangs indefinitely.
How do I stop this?
# function to take a single image called 'fileName' and place it in directory 'dir'
def takeImage(dir,fileName):
# calculate the view to capture to get the whole display window in.
clientRect = win32gui.GetClientRect(win32gui.GetForegroundWindow())
windowRect = win32gui.GetWindowRect(win32gui.GetForegroundWindow())
print(windowRect)
windowSize = [windowRect[2]-windowRect[0],windowRect[3]-windowRect[1]]
print(windowSize)
print(clientRect)
diffSize = [windowSize[0] -clientRect[2], windowSize[1] - clientRect[3]]
lrbBorder = diffSize[0]/2
topBorder = diffSize[1] - lrbBorder
print("sizeDiff = " + str(diffSize))
windowName = win32gui.GetWindowText(win32gui.GetForegroundWindow())
handleId = win32gui.GetForegroundWindow()
leftMar = designLabel.GetPosition()[0] + lrbBorder
topMar = designLabel.GetPosition()[1] + topBorder + designLabel.GetSize()[1]
rightMar = leftMar + scene.width
bottMar = topMar+scene.height
margins = [leftMar,topMar,rightMar,bottMar]
print(margins)
# now print the view.
#command_line = r"screenshot-cmd -wt '" + windowName + "' -rc " + str(margins[0]) + " " + str(margins[1]) + " " + str(margins[2]) + " " + str(margins[3]) + " -o " + fileName
command_line = r"screenshot-cmd -wt '" + windowName + "' -rc " + str(margins[0]) + " " + str(margins[1]) + " " + str(margins[2]) + " " + str(margins[3]) + " -o " + fileName
print(command_line)
args = shlex.split(command_line)
proc = subprocess.Popen(args)
proc.wait()
wx.Yield()
if not os.path.isdir(dir):
os.makedirs(dir)
newPath = os.path.join(dir,fileName)
if os.path.exists(newPath):
os.remove(newPath)
oldPath = os.path.join(os.getcwd(), fileName)
print("Old Path: " + oldPath)
print("Exists: " + str(os.path.exists(oldPath)))
shutil.move(oldPath,newPath)
return
#event called upon clicking 'takeTenImag' button
def takeTenImgE(evt):
global designNo
global workingDirectory
global numDesigns
fileNameRoot = "test_"
fileExtention = ".png"
# check there are at least 10 designs
if numDesigns > 9 and os.path.exists(workingDirectory):
# find directory path to put images in
dir = os.path.join(workingDirectory, "images")
# for each design
for x in range(10):
print("design =" + str(designNo))
fileName = fileNameRoot + str(designNo) + fileExtention
print("------------------")
print("for x = " + str(x) + " " + fileName)
# create image and save
print(dir)
takeImage(dir,fileName)
#move to next design
wx.PostEvent(forwardDesign, wx.CommandEvent(wx.wxEVT_COMMAND_BUTTON_CLICKED, forwardDesign.GetId()) )
wx.Yield()
print("design =" + str(designNo))
return
takeTenImg = wx.Button(p, label='Take Ten Images', pos=(rb + visScaleText.GetSize()[0]+10,takeImg.GetPosition()[1]+5 +takeImg.GetSize()[1]), size = (100,30))
takeTenImg.Bind(wx.EVT_BUTTON, takeTenImgE)
https://code.google.com/p/screenshot-cmd/

Barnaby, you may be over-complicating your subprocess use. Popen is typically used for when you need to communicate with the process during the time it is running. From the sound of it, you don't need to do that, so might want to use a higher level function. See the docs on subprocess' various invocations, and perhaps try using the call method. You'll need shell=True, as detailed in SO questions such as this one.

I've found that the error is in my calling of subprocess.
I was using:
command_line = r"screenshot-cmd -wt '" + windowName + ...."
args = shlex.split(command_line)
subprocess.call(args,shell=True)
changing this to:
command_line = r"screenshot-cmd -wt '" + windowName + ...."
subprocess.call(command_line,shell=True)
solves the hang.
What is peculiar is that both options work when not inside a wx button click event(i.e. a python script launched from the command line), but only the second works when inside the wx button click event.
If anyone could enlighten me why that would be most appreciated.
EDIT:
upon further investigation, the hang is caused by trying to specify the active window in screenshot-cmd.
To solve this I find the position of the window using
windowRect = win32gui.GetWindowRect(win32gui.GetForegroundWindow())
and then use screenshot-cmd without specifying a window.
This solves all issues although it is unclear why this causes problems