I created a pipeline with ValueProviders in order to use it as a template. However, I can't figure out how to use ValueProviders when testing the Pipeline. I can't just use values directly to test because my PTransforms are waiting for ValueProviders.
I do not know about Python, but in Java you may use StaticValue provider.
E.g. if you have the following interface for options:
interface BaseOptions extends DataflowPipelineOptions {
void setSource(ValueProvider<String> source);
ValueProvider<String> getSource();
}
Then you may use ValueProvider.StaticValueProvider.of(...) to initialise your parameter. Something like this:
BaseOptions options = PipelineOptionsFactory.fromArgs(args).as(BaseOptions.class);
options.setSource(ValueProvider.StaticValueProvider.of("/path/to/file"));
Pipeline p = Pipeline.create(options);
p.apply(TextIO.read().from(options.getSource()))
.apply("just print",
new ParDo().of(new DoFn<String, String>() {
#ProcessElement
public void processElement(ProcessContext c) {
System.out.println(c.element());
}
}));
p.run();
I put default values for the Value Providers in my pipeline opions :
class MypipelineOptions(PipelineOptions):
#classmethod
def _add_argparse_args(cls,parser):
parser.add_value_provider_argument('--variable',
type=float,
dest='variable',
default=5)
So that when I call MyPipelineOptions in the test file, it automatically uses the default values.
Related
I have been working on a problem for a while now which I cannot seem to resolve so I need some help! The problem is that I am writing a program in C# but I require a function from a Python file I created. This in itself is no problem:
...Usual Stuff
using IronPython.Hosting;
using IronPython.Runtime;
using Microsoft.Scripting;
using Microsoft.Scripting.Hosting;
namespace Program
{
public partial class Form1 : Form
{
Microsoft.Scripting.Hosting.ScriptEngine py;
Microsoft.Scripting.Hosting.ScriptScope s;
public Form1()
{
InitializeComponent();
py = Python.CreateEngine(); // allow us to run ironpython programs
s = py.CreateScope(); // you need this to get the variables
}
private void doPython()
{
//Step 1:
//Creating a new script runtime
var ironPythonRuntime = Python.CreateRuntime();
//Step 2:
//Load the Iron Python file/script into the memory
//Should be resolve at runtime
dynamic loadIPython = ironPythonRuntime.;
//Step 3:
//Invoke the method and print the result
double n = loadIPython.add(100, 200);
numericUpDown1.Value = (decimal)n;
}
}
}
However, this requires for the file 'first.py' to be wherever the program is once compiled. So if I wanted to share my program I would have to send both the executable and the python files which is very inconvenient. One way I thought to resolve this is by adding the 'first.py' file to the resources and running from there... but I don't know how to do this or even if it is possible.
Naturally the above code will not work for this as .UseFile method takes string arguments not byte[]. Does anyone know how I may progress?
Lets start with the simplest thing that could possibly work, you've got some code that looks a little like the following:
// ...
py = Python.CreateEngine(); // allow us to run ironpython programs
s = py.CreateScope(); // you need this to get the variables
var ironPythonRuntime = Python.CreateRuntime();
var x = py.CreateScriptSourceFromFile("SomeCode.py");
x.Execute(s);
var myFoo = s.GetVariable("myFoo");
var n = (double)myFoo.add(100, 200);
// ...
and we'd like to replace the line var x = py.CreateScriptSourceFromFile(... with something else; If we could get the embedded resource as a string, we could use ScriptingEngine.CreateScriptSourceFromString().
Cribbing this fine answer, we can get something that looks a bit like this:
string pySrc;
var resourceName = "ConsoleApplication1.SomeCode.py";
using (var stream = System.Reflection.Assembly.GetExecutingAssembly()
.GetManifestResourceStream(resourceName))
using (var reader = new System.IO.StreamReader(stream))
{
pySrc = reader.ReadToEnd();
}
var x = py.CreateScriptSourceFromString(pySrc);
I've got an Apache Beam pipeline that uses Java only firestore transforms in a python pipeline with the help of an expansion service that exposes the firestore read transforms.
Here I am trying to list all my documents in the collection named pokemon
This is my python pipeline:
def run():
"""Main function that defines pipeline and runs it."""
pipeline_options = get_pipeline_options(**vars(args))
request = ListDocumentsRequest()
request.parent = 'projects/PROJECT_ID/databases'
request.collection_id = 'pokemon'
with beam.Pipeline(options=pipeline_options) as pipeline:
_ = (pipeline
| 'Create Requests' >> beam.Create([request.transaction])
.with_output_types(typing.List[bytes])
| 'Read from JavaFirestore' >> beam.ExternalTransform(
'my.beam.transform.firestore_list_documents',
ImplicitSchemaPayloadBuilder({'parent': request.parent,
'collectionId': request.collection_id}),
"localhost:12345")
| 'Write' >> WriteToText('output_files/output')
)
This is the Java code that exposes the transform/URN in the expansion service:
public static class FirestoreListDocumentsBuilder implements
ExternalTransformBuilder<FirestoreTransformsConfiguration, PCollection<ListDocumentsRequest>, PCollection<Document>> {
#Override
public PTransform<PCollection<ListDocumentsRequest>, PCollection<Document>> buildExternal(
FirestoreTransformsConfiguration configuration) {
return FirestoreIO.v1().read().listDocuments().build();
}
}
#AutoService(ExternalTransformRegistrar.class)
public class FirestoreTransformsRegistrar implements ExternalTransformRegistrar {
final static String URN_LIST_DOCS = "my.beam.transform.firestore_list_documents";
#Override
public Map<String, ExternalTransformBuilder<?, ?, ?>> knownBuilderInstances() {
return ImmutableMap.of(
URN_LIST_DOCS,new FirestoreListDocumentsBuilder()
);
}
}
But I am getting the following error when running the python pipeline this:
java.lang.IllegalArgumentException: Unknown Coder URN beam:coder:pickled_python:v1. Known URNs: [beam:coder:avro:generic:v1, beam:coder:bytes:v1, beam:coder:bool:v1, beam:coder:string_utf8:v1, beam:coder:kv:v1, beam:coder:varint:v1, beam:coder:interval_window:v1, beam:coder:iterable:v1, beam:coder:timer:v1, beam:coder:length_prefix:v1, beam:coder:global_window:v1, beam:coder:windowed_value:v1, beam:coder:param_windowed_value:v1, beam:coder:double:v1, beam:coder:row:v1, beam:coder:sharded_key:v1, beam:coder:custom_window:v1]
I am struggling to find the solution to this problem, anyone that could help me out?
This is because (surprisingly) List is not a "standard" cross-language coder. Try using with_output_types(typing.Iterable[bytes]) instead. (Your java code will have to change to take a PCollection of Iterable<byte[]> to match.)
For some additional information, please take a look on this page.
I am looking for the matplotlib set plane color equivalent in Bokeh. I generated the image by wrapping TS code into Python. I received the code from the example section of Bokeh API Doc. All options are currently in their default mode. I tried playing around with various options such as: Background, showGrayColor, showShadow etc but nothing seems to get rid of the shadow. Any help would be appreciated!
The image I have looks like this:
I want the background to be white instead of the shadow currently visible. Anyone know how to do this?
This is the code I used:
TS_CODE1 = """
// This custom model wraps one part of the third-party vis.js library:
//
// http://visjs.org/index.html
//
// Making it easy to hook up python data analytics tools (NumPy, SciPy,
// Pandas, etc.) to web presentations using the Bokeh server.
import {LayoutDOM, LayoutDOMView} from "models/layouts/layout_dom"
import {ColumnDataSource} from "models/sources/column_data_source"
import {LayoutItem} from "core/layout"
import * as p from "core/properties"
declare namespace vis {
class Graph3d {
constructor(el: HTMLElement, data: object, OPTIONS: object)
setData(data: vis.DataSet): void
}
class DataSet {
add(data: unknown): void
}
}
// This defines some default options for the Graph3d feature of vis.js
// See: http://visjs.org/graph3d_examples.html for more details.
const OPTIONS = {
width: '500px',
height: '500px',
style: 'surface',
showPerspective: false,
showGrid: true,
keepAspectRatio: true,
verticalRatio: 1.0,
legendLabel: 'stuff',
cameraPosition: {
horizontal:2,
vertical: 0.5,
distance: 1.8,
},
}
// To create custom model extensions that will render on to the HTML canvas
// or into the DOM, we must create a View subclass for the model.
//
// In this case we will subclass from the existing BokehJS ``LayoutDOMView``
export class Surface3d1View extends LayoutDOMView {
model: Surface3d1
private _graph: vis.Graph3d
initialize(): void {
super.initialize()
const url = "https://cdnjs.cloudflare.com/ajax/libs/vis/4.16.1/vis.min.js"
const script = document.createElement("script")
script.onload = () => this._init()
script.async = false
script.src = url
document.head.appendChild(script)
}
private _init(): void {
// Create a new Graph3s using the vis.js API. This assumes the vis.js has
// already been loaded (e.g. in a custom app template). In the future Bokeh
// models will be able to specify and load external scripts automatically.
//
// BokehJS Views create <div> elements by default, accessible as this.el.
// Many Bokeh views ignore this default <div>, and instead do things like
// draw to the HTML canvas. In this case though, we use the <div> to attach
// a Graph3d to the DOM.
this._graph = new vis.Graph3d(this.el, this.get_data(), OPTIONS)
// Set a listener so that when the Bokeh data source has a change
// event, we can process the new data
this.connect(this.model.data_source.change, () => {
this._graph.setData(this.get_data())
})
}
// This is the callback executed when the Bokeh data has an change. Its basic
// function is to adapt the Bokeh data source to the vis.js DataSet format.
get_data(): vis.DataSet {
const data = new vis.DataSet()
const source = this.model.data_source
for (let i = 0; i < source.get_length()!; i++) {
data.add({
x: source.data[this.model.x][i],
y: source.data[this.model.y][i],
z: source.data[this.model.z][i],
})
}
return data
}
get child_models(): LayoutDOM[] {
return []
}
_update_layout(): void {
this.layout = new LayoutItem()
this.layout.set_sizing(this.box_sizing())
}
}
// We must also create a corresponding JavaScript BokehJS model subclass to
// correspond to the python Bokeh model subclass. In this case, since we want
// an element that can position itself in the DOM according to a Bokeh layout,
// we subclass from ``LayoutDOM``
export namespace Surface3d1 {
export type Attrs = p.AttrsOf<Props>
export type Props = LayoutDOM.Props & {
x: p.Property<string>
y: p.Property<string>
z: p.Property<string>
data_source: p.Property<ColumnDataSource>
}
}
export interface Surface3d1 extends Surface3d1.Attrs {}
export class Surface3d1 extends LayoutDOM {
properties: Surface3d1.Props
__view_type__: Surface3d1View
constructor(attrs?: Partial<Surface3d1.Attrs>) {
super(attrs)
}
// The ``__name__`` class attribute should generally match exactly the name
// of the corresponding Python class. Note that if using TypeScript, this
// will be automatically filled in during compilation, so except in some
// special cases, this shouldn't be generally included manually, to avoid
// typos, which would prohibit serialization/deserialization of this model.
static __name__ = "Surface3d1"
static init_Surface3d1() {
// This is usually boilerplate. In some cases there may not be a view.
this.prototype.default_view = Surface3d1View
// The #define block adds corresponding "properties" to the JS model. These
// should basically line up 1-1 with the Python model class. Most property
// types have counterparts, e.g. ``bokeh.core.properties.String`` will be
// ``p.String`` in the JS implementatin. Where the JS type system is not yet
// as rich, you can use ``p.Any`` as a "wildcard" property type.
this.define<Surface3d1.Props>({
x: [ p.String ],
y: [ p.String ],
z: [ p.String ],
data_source: [ p.Instance ],
})
}
}
"""
# This custom extension model will have a DOM view that should layout-able in
# Bokeh layouts, so use ``LayoutDOM`` as the base class. If you wanted to create
# a custom tool, you could inherit from ``Tool``, or from ``Glyph`` if you
# wanted to create a custom glyph, etc.
class Surface3d1(LayoutDOM):
# The special class attribute ``__implementation__`` should contain a string
# of JavaScript code that implements the browser side of the extension model.
__implementation__ = TypeScript(TS_CODE1)
# Below are all the "properties" for this model. Bokeh properties are
# class attributes that define the fields (and their types) that can be
# communicated automatically between Python and the browser. Properties
# also support type validation. More information about properties in
# can be found here:
#
# https://docs.bokeh.org/en/latest/docs/reference/core/properties.html#bokeh-core-properties
# This is a Bokeh ColumnDataSource that can be updated in the Bokeh
# server by Python code
data_source = Instance(ColumnDataSource)
# The vis.js library that we are wrapping expects data for x, y, and z.
# The data will actually be stored in the ColumnDataSource, but these
# properties let us specify the *name* of the column that should be
# used for each field.
x = String
y = String
z = String
surface1 = Surface3d1(x="x", y="y", z="z", data_source=source, width=600, height=600)
indent preformatted text by 4 spaces
I have to check with a Python box if a word I've told to Pepper (saved externally from a dialog box) is inside a list (created as string,and saved into ALMemory from SSH in Matlab), and do something if yes or not.
How can I do this?
def onInput_onStart(self):
#self.onStopped() #activate the output of the box
picklist = ALProxy("ALMemory")
list=picklist.getData("myFood")
def food(self):
if food in list:
tts=ALProxy("ALDialog")
tts.say("Available")
I personally would just manage it on the web using js, when it comes to this kind of stuff boxes give more trouble than it's worth. Raise an event with the string you want and check if the word is inside the list. After that you can either use the tts (as you seem to be trying to do) or raise and event (sending the true/false as a parameter) and use it to trigger whatever you want.
Javascript:
session = null
QiSession(connected, disconnected, location.host);
tts = null;
function connected(s) {
console.log("Session connected");
session = s;
startSubscribe();
session.service("ALTextToSpeech").then(function (t) {
tts = t;
});
}
function disconnected(error) {
console.log("Session disconnected");
}
function startSubscribe() {
session.service("ALMemory").then(function (memory) {
memory.subscriber("toTablet").then(function (subscriber) {
subscriber.signal.connect(functionThatChecks)
});
});
}
function functionThatChecks(word)
{
tts.stopAll();
/*Check if exists*/
tts.say("It exists"); //Or raise an event
}
Dialog
u: (word) $eventName="word"
Choregraphe
You need to use self.list before other functions can access it.
You also need to pass users_food to the function when you call food().
Assuming that list is a list of strings, and users_food is a string.
def onInput_onStart(self):
#self.onStopped() #activate the output of the box
picklist = ALProxy("ALMemory")
self.list=picklist.getData("myFood")
def food(self, users_food):
if users_food in self.list:
tts=ALProxy("ALDialog")
tts.say("Available")
i want to pass xml format data into python from flex.i know how to pass from flex but my question is how can i get the passed data in python and then the data should be inserted into mysql.and aslo i want to retrieve the mysql data to the python(cgi),the python should convert all the data into xml format,and pass all the data to the flex..
Thank's in advance.....
See http://www.artima.com/weblogs/viewpost.jsp?thread=208528 for more details, here is a breif overview of what I think you are looking for.
The SimpleXMLRPCServer library allows you to easily create a server. Here's about the simplest server you can create, which provides two services to manipulate strings:
import sys
from random import shuffle
from SimpleXMLRPCServer import SimpleXMLRPCServer
class MyFuncs:
def reverse(self, str) :
x = list(str);
x.reverse();
return ''.join(x);
def scramble(self, str):
x = list(str);
shuffle(x);
return ''.join(x);
server = SimpleXMLRPCServer(("localhost", 8000))
server.register_instance(MyFuncs())
server.serve_forever()
Once you make a connection to the server, that server acts like a local object. You call the server's methods just like they're ordinary methods of that object.
This is about as clean an RPC implementation as you can hope for (and other Python RPC libraries exist; for example, CORBA clients). But it's all text based; not very satisfying when trying to create polished applications with nice GUIs. What we'd like is the best of all worlds -- Python (or your favorite language) doing the heavy lifting under the covers, and Flex creating the user experience.
To use the library, download it and unpack it somewhere. The package includes all the source code and the compiled as3-rpclib.swc library -- the .swc extension indicates an archive file, and pieces of this library can be pulled out and incorporated into your final .swf. To include the library in your project, you must tell Flexbuilder (you can get a free trial or just use the free command-line tools, and add on the Apollo portion) where the library is located by going to Project|Properties and selecting "Apollo Build Path," then choosing the "Library path" tab and pressing the "Add SWC..." button. Next, you add the namespace ak33m to your project as seen in the code below, and you're ready to create an XMLRPCObject.
Note: the only reason I used Apollo here was that I was thinking in terms of desktop applications with nice UIs. You can just as easily make it a Flex app.
Here's the entire Apollo application as a single MXML file, which I'll explain in detail:
<?xml version="1.0" encoding="utf-8"?>
<mx:ApolloApplication xmlns:mx="http://www.adobe.com/2006/mxml"
xmlns:ak33m="http://ak33m.com/mxml" layout="absolute">
<mx:Form>
<mx:FormHeading label="String Modifier"/>
<mx:FormItem label="Input String">
<mx:TextInput id="instring" change="manipulate()"/>
</mx:FormItem>
<mx:FormItem label="Reversed">
<mx:Text id="reversed"/>
</mx:FormItem>
<mx:FormItem label="Scrambled">
<mx:Text id="scrambled"/>
</mx:FormItem>
</mx:Form>
<ak33m:XMLRPCObject id="server" endpoint="http://localhost:8000"/>
<mx:Script>
<![CDATA[
import mx.rpc.events.ResultEvent;
import mx.rpc.events.FaultEvent;
import mx.rpc.AsyncToken;
import mx.controls.Alert;
import mx.collections.ItemResponder;
private function manipulate() : void {
server.reverse(instring.text).addResponder(new ItemResponder(reverseResult, onFault));
server.scramble(instring.text).addResponder(new ItemResponder(scrambleResult, onFault));
}
private function reverseResult(event : ResultEvent, token : AsyncToken = null) : void {
reversed.text = event.result.toString();
}
private function scrambleResult(event : ResultEvent, token : AsyncToken = null) : void {
scrambled.text = event.result.toString();
}
private function onFault (event : FaultEvent, token : AsyncToken = null) : void {
Alert.show(event.fault.faultString, event.fault.faultCode);
}
]]>
</mx:Script>
</mx:ApolloApplication>