Develop Reference

show user 'processing' animation in python

i just wrote an extremely long script that takes a list of URLs and runs a series of API calls, processes them etc.
Unfortunately for majority of the calls, response is empty (to give scale, i get result for ~100 / 4000 urls )
I'm wondering if there is a way to show in terminal (or powershell) some kind of animation to indicate that the process is still running. Currently - i have the standard blinking underscore.
I was thinking of something like alterating between \ - / |
i definitely don't want to trash the console with printing too much.
does anyone know if something like that is possible ?

This is one way you can do it in PowerShell.
$progress = '\', '|', '/', '-'
$i = 0
[console]::Write('Doing something...')
while($true) {
$char = $progress[$i++ % $progress.Count]
[console]::Write("$char$([char]0x8)")
Start-Sleep -Milliseconds 300
}
Relevant docs:
Console.Write(String)
Modulus %
Index operator [ ]

Leveraging Spell Checker on local machine?

I notice that common applications on a given machine (Mac, Linux, or Windows) have their respective spell checkers. Everything from various IDE, to MS Word/Office, to Note taking software.
I am trying to utilize the built in utility of our respective machines in order to analyze strings for syntactic correctness. It seems that I cant just use what is on the machine and would have to likely download a dictionary in which to compare against.
I was not sure if there was a better way to accomplish this. I was looking at trying to do things locally, but I was not opposed to doing api or curl requests to determine if the words in a string are spelled correctly.
I was looking at:
LanguageTool (hello wrold failed to return an error)
Google's tbproxy seems to not be functional
Dictionary / Meriam-Webster require api keys for automation.
I was looking at Node packages and noticed spell checker modules which encapsulate wordlists as well.
Is there a way to utilize the built in machine dictionaries at all, or would it be ideal if I download a dictionary / wordlist to compare against?
I am thinking a wordlist might be best bet, but i didnt want to reinvent the wheel. What have others done to accomplish similar?

The Credit is going to Lukas Knuth. I want to give an explicit how to for using dictionary and nspell.
Install The following 2 dependancies:
npm install nspell dictionary-en-us
Here is a Sample File I wrote in order to solve the problem.
// Node File
// node spellcheck.js [path]
// path: [optional] either absolute or local path from pwd/cwd
// if you run the file from within Seg.Ui.Frontend/ it works as well.
// node utility/spellcheck.js
// OR from the utility directory using a path:
// node spellcheck.js ../src/assets/i18n/en.json
var fs = require("fs");
var dictionary = require("dictionary-en-us");
var nspell = require("nspell");
var process = require("process");
// path to use if not defined.
var path = "src/assets/i18n/en.json"
let strings = [];
function getStrings(json){
let keys = Object.keys(json);
for (let idx of keys){
let val = json[idx];
if (isObject(val)) getStrings(val);
if (isString(val)) strings.push(val)
}
}
function sanitizeStrings(strArr){
let set = new Set();
for (let sentence of strArr){
sentence.split(" ").forEach(word => {
word = word.trim().toLowerCase();
if (word.endsWith(".") || word.endsWith(":") || word.endsWith(",")) word = word.slice(0, -1);
if (ignoreThisString(word)) return;
if (word == "") return;
if (isNumber(word)) return;
set.add(word)
});
}
return [ ...set ];
}
function ignoreThisString(word){
// we need to ignore special cased strings, such as items with
// Brackets, Mustaches, Question Marks, Single Quotes, Double Quotes
let regex = new RegExp(/[\{\}\[\]\'\"\?]/, "gi");
return regex.test(word);
}
function spellcheck(err, dict){
if (err) throw err;
var spell = nspell(dict);
let misspelled_words = strings.filter( word => {
return !spell.correct(word)
});
misspelled_words.forEach( word => console.log(`Plausible Misspelled Word: ${word}`))
return misspelled_words;
}
function isObject(obj) { return obj instanceof Object }
function isString(obj) { return typeof obj === "string" }
function isNumber(obj) { return !!parseInt(obj, 10)}
function main(args){
//node file.js path
if (args.length >= 3) path = args[2]
if (!fs.existsSync(path)) {
console.log(`The path does not exist: ${process.cwd()}/${path}`);
return;
}
var content = fs.readFileSync(path)
var json = JSON.parse(content);
getStrings(json);
// console.log(`String Array (length: ${strings.length}): ${strings}`)
strings = sanitizeStrings(strings);
console.log(`String Array (length: ${strings.length}): ${strings}\n\n`)
dictionary(spellcheck);
}
main(process.argv);
This will return a subset of strings to look at and they may be misspelled or false positives.
A false positive will be denoted as:
An acronym
non US English variants for words
Un recognized Proper Nouns, Days of the Week and Months for example.
Strings which contain parenthese. This can be augmented out by trimming them off the word.
Obviously, this isnt for all cases, but i added an ignore this string function you can leverage if say it contains a special word or phrase the developers want ignored.
This is meant to be run as a node script.

Your question is tagged as both NodeJS and Python. This is the NodeJS specific part, but I imagine it's very similar to python.
Windows (from Windows 8 onward) and Mac OS X do have built-in spellchecking engines.
Windows: The "Windows Spell Checking API" is a C/C++ API. To use it with NodeJS, you'll need to create a binding.
Mac OS X: "NSSpellChecker" is part of AppKit, used for GUI applications. This is an Objective-C API, so again you'll need to create a binding.
Linux: There is no "OS specific" API here. Most applications use Hunspell but there are alternatives. This again is a C/C++ library, so bindings are needed.
Fortunately, there is already a module called spellchecker which has bindings for all of the above. This will use the built-in system for the platform it's installed on, but there are multiple drawbacks:
1) Native extensions must be build. This one has finished binaries via node-pre-gyp, but these need to be installed for specific platforms. If you develop on Mac OS X, run npm install to get the package and then deploy your application on Linux (with the node_modules-directory), it won't work.
2) Using build-in spellchecking will use defaults dictated by the OS, which might not be what you want. For example, the used language might be dictated by the selected OS language. For a UI application (for example build with Electron) this might be fine, but if you want to do server-side spellchecking in languages other than the OS language, it might prove difficult.
At the basic level, spellchecking some text boils down to:
Tokenizing the string (e.g. by spaces)
Checking every token against a list of known correct words
(Bonus) Gather suggestions for wrong tokens and give the user options.
You can write part 1 yourself. Part 2 and 3 require a "list of known correct words" or a dictionary. Fortunately, there is a format and tools to work with it already:
simple-spellchecker can work with .dic-files.
nspell is a JS implementation of Hunspell, complete with its own dictionary packages.
Additional Dictionaries can be found for example in this repo
With this, you get to choose the language, you don't need to build/download any native code and your application will work the same on every platform. If you're spellchecking on the server, this might be your most flexible option.

How to use a variable in API payload in Jupyter Notebook

I'm using Jupyter Notebook to gather up a bunch of API calls then use various plotting tools to generate pretty graphs.
I have the notebook working great when using constants in the API payload. Now I'm trying to replace the constants with user entered variables and having some challenges.
The following method for variable use works when the variable isn't part of a filter or function:
key = input('Enter your subscription API Key then press Enter: ')
When prompted, the user enters the key and that value is used perfectly fine in the following:
headers = {
'X-Co-Integration-Key': key,
'Content-Type': "application/json",
'Cache-Control': "no-cache",
}
Next, enter another variable for a guide called guideId:
guideId = input('Enter the Guide ID of the guide for which you want statistics then press Enter: ')
When prompted, the user enters the guideId and that value is stored, but I can't seem to get it to be used properly in the API payload.
I've tried several different methods of inserting the variable and the following gets me the closest to something that works:
{\n \"filter\": \"id== \"" + guideId + "\n }
The API call runs, but I get the following error:
{"overall": {"DeserializationError": "invalid character 'q' after object key:value pair"}, "fields": {"pipeline": "Required"}}
It looks like it is reading the variable and stopping after it hits the first character in the variable, which, in this case is a q.
I tried changing the variable to start with a number. No change in behavior.
I tried using str(guideId) - no change in behavior.
I'm stumped. Any ideas?

Printing in Duplexpage a word document

I am trying to automate the task of printing two copies at double page of ~30 Word document (*.doc). I want to send the program converted to .exe (I plan it just for Windows computers) using py2exe. I know that I can manually check the options but I will not be able to do so on the 20 or so computer where it will be used, as well as I cannot install in this computers new software (That's why I want to convert it into .exe).
I copied this solution to print, but I can't adapt it to do what I want:
from win32com import client
import time
word = client.Dispatch("Word.Application")
filename=input("What files do you want to print?")
def printWordDocument(filename):
"""Given a name of a file prints it. TODO: Add double page."""
word.Documents.Open(filename)
word.ActiveDocument.PrintOut()
time.sleep(2)
word.ActiveDocument.Close()
word.Quit()
I couldn't find any option to print in double pages, or at least automatically, the only option of double page of PrintOut method is ManualDuplexPrint which in the documentation says: "True to print a two-sided document on a printer without a duplex printing kit.", but I don't want to make it even easier to print all the set of documents. And make a program portable to other computers, without modifying the Word documents (I don't create them).
Any other way to do it? Or any other option to do it?
UPDATE
I am not able to code in visual-basic (yet), but if I get a template or some hints I think I will manage to make something adapted to my conditions.

I have ended doing a macro, but this just works for my own computer and not for all the computers where should work.
Sub Test()
'
' Test Macro
' Print in double page and 2 copies
'
ActivePrinter = "Xerox WC 24 PCL"
Application.PrintOut FileName:="", Range:=wdPrintAllDocument, Item:= _
wdPrintDocumentWithMarkup, Copies:=2, Pages:="", PageType:= _
wdPrintAllPages, Collate:=True, Background:=True, PrintToFile:=False, _
PrintZoomColumn:=0, PrintZoomRow:=0, PrintZoomPaperWidth:=0, _
PrintZoomPaperHeight:=0
End Sub

ruby fetching url content is always empty

I am so frustrated trying to use Ruby to fetch a specific url content.
I've tried many different ways like open-uri, standard request none worked so far. I always get empty html. I also tried to use python to fetch the same url which always returned the correct html content. I am really not sure why... Please help as I am newbiew to both Ruby and Python... I want to use Ruby (prefer the tidy syntax and human friendly function names, easier to install libs using gem and homebrew (on mac) than python easy_install) but I am now considering Python because it just works (yet still trying to get my head around 2.x and 3.x issue). I may be doing something really stupid but I think is very unlikely.
ruby 1.9.2p136 (2010-12-25 revision 30365) [i386-darwin10.6.0]
Implementation 1:
url = URI.parse('http//:www.stackoverflow.com/') req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) {|http| http.request(req) }
puts res.body #empty
Implementation 2:
doc = Nokogiri::HTML(open("http//:www.stackoverflow.com/", "User-Agent" => "Safari"))
#empty
#I tried to use without user agent, without Nokogiri none worked.
Python Implementation which worked every time perfectly
f = urllib.urlopen("http//:www.stackoverflow.com/")
# Read from the object, storing the page's contents in 's'.
s = f.read()
f.close()
print s

If that is your exact code it is invalid for several reasons.
http: should be http://
URL needs a path. if you want the root page of example.com it needs to be http://example.com/ the trailing slash is significant.
if you put 2 lines of code on one line you need to use ; to denote the end of the first line
SO
require 'net/http'
url = URI.parse('http://www.yellowpages.com.au/search/listings?clue=plumber&locationClue=Australia')
req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) {|http| http.request(req) }
puts res.body
Same is true with using open in nokogiri
EDIT: that site is returning bad results many times:
counter = 0
20.times do
url = URI.parse('http://www.yellowpages.com.au/search/listings?clue=plumber&locationClue=Australia')
req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) {|http| http.request(req) }
sleep 1
counter +=1 unless res.body.empty?
end
puts counter
for me this only returned once a non empty body. If you substitute in another site it works all the time
curl "http://www.yellowpages.com.au/search/listings?clue=plumber&locationClue=Australia"
Yields the same inconsistent results.

Two examples with openURI (standard lib), a wrapper for (among others) the rather cumbersome Net::HTTP :
require 'open-uri'
open("http://www.stackoverflow.com/"){|f| puts f.read}
puts URI::parse("http://www.google.com/").read