Develop Reference

Flink Python Datastream API Kafka Consumer

Im new to pyflink. Im tryig to write a python program to read data from kafka topic and prints data to stdout. I followed the link Flink Python Datastream API Kafka Producer Sink Serializaion. But i keep seeing NoSuchMethodError due to version mismatch. I have added the flink-sql-kafka-connector available at https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.11/1.13.0/flink-sql-connector-kafka_2.11-1.13.0.jar. Can someone help me in with a proper example to do this? Following is my code
import json
import os
from pyflink.common import SimpleStringSchema
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.connectors import FlinkKafkaConsumer
from pyflink.common.typeinfo import Types
def my_map(obj):
json_obj = json.loads(json.loads(obj))
return json.dumps(json_obj["name"])
def kafkaread():
env = StreamExecutionEnvironment.get_execution_environment()
env.add_jars("file:///automation/flink/flink-sql-connector-kafka_2.11-1.10.1.jar")
deserialization_schema = SimpleStringSchema()
kafkaSource = FlinkKafkaConsumer(
topics='test',
deserialization_schema=deserialization_schema,
properties={'bootstrap.servers': '10.234.175.22:9092', 'group.id': 'test'}
)
ds = env.add_source(kafkaSource).print()
env.execute('kafkaread')
if __name__ == '__main__':
kafkaread()
But python doesnt recognise the jar file and throws the following error.
Traceback (most recent call last):
File "flinkKafka.py", line 31, in <module>
kafkaread()
File "flinkKafka.py", line 20, in kafkaread
kafkaSource = FlinkKafkaConsumer(
File "/automation/flink/venv/lib/python3.8/site-packages/pyflink/datastream/connectors.py", line 186, in __init__
j_flink_kafka_consumer = _get_kafka_consumer(topics, properties, deserialization_schema,
File "/automation/flink/venv/lib/python3.8/site-packages/pyflink/datastream/connectors.py", line 336, in _get_kafka_consumer
j_flink_kafka_consumer = j_consumer_clz(topics,
File "/automation/flink/venv/lib/python3.8/site-packages/pyflink/util/exceptions.py", line 185, in wrapped_call
raise TypeError(
TypeError: Could not found the Java class 'org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer'. The Java dependencies could be specified via command line argument '--jarfile' or the config option 'pipeline.jars'
What is the correct location to add the jar file?

I see that you downloaded flink-sql-connector-kafka_2.11-1.13.0.jar, but the code loades flink-sql-connector-kafka_2.11-1.10.1.jar.
May be you can have a check

just need to check the path to flink-sql-connector jar

You should add jar file of flink-sql-connector-kafka, it depends on your pyflink and scala version. If versions are true, check your path in add_jars function if the jar package is here.

google ads api - "argument should be integer or bytes-like object, not 'str'"

I've been trying to follow the examples and documentation for the python ad_manager library for the google ads API, but I haven't been able to complete a successful request. I currently have my developer token, client_id, client_secret, and refresh_token in my google ads YAML file, but I'm constantly getting the error "argument should be integer or bytes-like object, not 'str'" when calling the function WaitForReport following the example code below. I was wondering if anyone had any advice on how I could tackle this issue.
import tempfile
# Import appropriate modules from the client library.
from googleads import ad_manager
from googleads import errors
def main(client):
# Initialize a DataDownloader.
report_downloader = client.GetDataDownloader(version='v202111')
# Create report job.
report_job = {
'reportQuery': {
'dimensions': ['COUNTRY_NAME', 'LINE_ITEM_ID', 'LINE_ITEM_NAME'],
'columns': ['UNIQUE_REACH_FREQUENCY', 'UNIQUE_REACH_IMPRESSIONS',
'UNIQUE_REACH'],
'dateRangeType': 'REACH_LIFETIME'
}
}
try:
# Run the report and wait for it to finish.
report_job_id = report_downloader.WaitForReport(report_job)
except errors.AdManagerReportError as e:
print('Failed to generate report. Error was: %s' % e)
# Change to your preferred export format.
export_format = 'CSV_DUMP'
report_file = tempfile.NamedTemporaryFile(suffix='.csv.gz', delete=False)
# Download report data.
report_downloader.DownloadReportToFile(
report_job_id, export_format, report_file)
report_file.close()
# Display results.
print('Report job with id "%s" downloaded to:\n%s' % (
report_job_id, report_file.name))
if __name__ == '__main__':
# Initialize client object.
ad_manager_client = ad_manager.AdManagerClient.LoadFromStorage()
main(ad_manager_client)
Edit:
Below is the stack trace:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/googleads/common.py", line 984, in MakeSoapRequest
return soap_service_method(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zeep/proxy.py", line 46, in __call__
return self._proxy._binding.send(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 135, in send
return self.process_reply(client, operation_obj, response)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 229, in process_reply
return self.process_error(doc, operation)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 317, in process_error
raise Fault(
zeep.exceptions.Fault: Unknown fault occured
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "google_ads.py", line 72, in <module>
main(ad_manager_client)
File "google_ads.py", line 33, in main1
report_job_id = report_downloader.WaitForReport(report_job)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/googleads/ad_manager.py", line 784, in WaitForReport
report_job_id = service.runReportJob(report_job)['id']
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/googleads/common.py", line 989, in MakeSoapRequest
underlying_exception = e.detail.find(
TypeError: argument should be integer or bytes-like object, not 'str'

In your YAML file, do you have your account number in quotes? (either single or double?)
Additionally, I would highly recommend not going with this API if you have the option. It will be sunset in April and will no longer work. The newer google ads API (as opposed to the AdWords API) is available, stable and much easier to work with. The ad manager examples are good too.

The problem seems to be that zeep raises a WebFault which includes the returned XML response as a string in zeep.Fault.detail.
Somewhat counter-intuitive, this attribute is not a string, but a bytes sequence because zeep.wsdl.utils.etree_to_string calls etree.tostring() with encoding="utf-8" instead of encoding="unicode"—the latter would make sure it's a proper string.
googleads then tries to look for specific error strings inside the XML using find(), but even though find() is defined both on str and bytes, the type of the substring to look for needs to align.
Thus, in
underlying_exception = e.detail.find(
'{%s}ApiExceptionFault' % self._GetBindingNamespace())
bytes.find() is called with a str argument, causing the ValueError you experience.
I'd argue that zeep.wsdl.utils.etree_to_string() should be adjusted to actually return a str instead of bytes. You could try opening an issue on Zeep's Github repository.

Missing request - Pact python

I'm new to pact and I've understood the concept but having hard time understanding and implementing the code.
Here I'm trying to do a simple pact for get_users from reqres.in.
I believe the first (pact ... code does the mock provider part and I compare that using the pact.json file.
import os
import requests
import pytest
from pact import Consumer, Provider, Format
import unittest
import json
pact = Consumer('Consumer').has_pact_with(Provider('Provider'), port=1234, host_name='localhost')
pact.start_service()
CURR_FILE_PATH = os.path.dirname(os.path.abspath(__file__))
PACT_DIR = os.path.join(CURR_FILE_PATH, '')
PACT_FILE = os.path.join(PACT_DIR, 'pact.json')
#defining class
class GetUsers(unittest.TestCase):
def test_get_board(self):
with open(os.path.join(PACT_DIR, PACT_FILE), 'rb') as path_file:
pact_file_json = json.load(path_file)
print('pact_json')
(pact
.given('Request to send message')
.upon_receiving('a request for response or send message')
.with_request(method = 'GET', path = '/api/users?page=2')
.will_respond_with(status = 200, body = pact_file_json))
with pact:
result = requests.get('http://reqres.in/api/users?page=2')
print('actual response')
self.assertEqual(pact_file_json, result.json())
pact.verify()
ge = GetUsers()
print(ge.test_get_board())
however, when I run the code, I get the following error
the data do not match but I verified it in another code.
Traceback (most recent call last):
File "C:\Users\Desktop\Pact\contract_test.py", line 41, in <module>
print(ge.test_get_board())
File "C:\Users\Desktop\Pact\contract_test.py", line 34, in test_get_board
print(data)
File "C:\Users\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pact\pact.py", line 370, in __exit__
self.verify()
File "C:\Users\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pact\pact.py", line 269, in verify
assert resp.status_code == 200, resp.text
AssertionError: Actual interactions do not match expected interactions for mock MockService.
says missing requests
Missing requests:
GET https://reqres.in/api/users?page=2

I'm not sure what you're trying to attempt here - why are you opening up a pact.json file in this context? The consumer pact package will automatically serialise any contracts if successful, and you don't seem to be reading the file in anywhere.
The second problem seems to be that some code somewhere, is issuing (or requesting?) the query string page=2 but the code you've shown doesn't make use of any query strings.
Could you have another mock service running somewhere?
Examples
https://docs.pactflow.io/docs/examples/python/consumer/readme
https://github.com/pact-foundation/pact-python/blob/master/examples/e2e/tests/consumer/test_user_consumer.py

How to save a presentation to a file-like object in Python 3

Python 3 replaced StringIO.StringIO with io.StringIO. I've been able to successfully save presentations using the former, but it doesn't appear to work for the latter.
from pptx import Presentation
from io import StringIO
presentation = Presentation('presentation.pptx')
output = StringIO()
presentation.save(output)
The above code produces:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\site-packages\pptx\presentation.py", line 46, in save
self.part.save(file)
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\site-packages\pptx\parts\presentation.py", line 118, in save
self.package.save(path_or_stream)
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\site-packages\pptx\opc\package.py", line 166, in save
PackageWriter.write(pkg_file, self.rels, self.parts)
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\site-packages\pptx\opc\pkgwriter.py", line 33, in write
PackageWriter._write_content_types_stream(phys_writer, parts)
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\site-packages\pptx\opc\pkgwriter.py", line 47, in _write_content_types_stream
phys_writer.write(CONTENT_TYPES_URI, content_types_blob)
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\site-packages\pptx\opc\phys_pkg.py", line 156, in write
self._zipf.writestr(pack_uri.membername, blob)
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\zipfile.py", line 1645, in writestr
with self.open(zinfo, mode='w') as dest:
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\zipfile.py", line 1349, in open
return self._open_to_write(zinfo, force_zip64=force_zip64)
File "C:\Users\mgplante\AppData\Local\Continuum\Anaconda2\envs\ppt_gen\lib\zipfile.py", line 1462, in _open_to_write
self.fp.write(zinfo.FileHeader(zip64))
TypeError: string argument expected, got 'bytes'
Is there a way to save a presentation to to a file-like object in Python 3, or am I going to have to use Python 2 for this project?

How about BytesIO()?
from pptx import Presentation
from io import BytesIO
presentation = Presentation('presentation.pptx')
output = BytesIO()
presentation.save(output)
output.seek(0)
# from here do what you like with output, e.g. pass it to something expecting bytes with output.read()
This removes the error at least.

Hannu's answer is quite right, and is precisely the code that is used to verify this behavior in the test suite for python-pptx:
stream = BytesIO()
presentation.save(stream)
https://github.com/scanny/python-pptx/blob/master/features/steps/presentation.py#L105
If that code is giving you a blank presentation, then something else is going on. I would reproduce that behavior, get it stable and repeatable, then ask the question to the effect "Why am I getting a blank presentation?" in another SO question, posting with it the full minimum code that gives you that behavior.
This is the second time I've heard of something like this, which makes me suspect there's actually something half-way systematic happening under the covers to produce this behavior. But at the same time, it's extremely unlikely that you would end up with a fully working presentation, just empty of slides, as a partial failure of attempting a save to a stream.
A common situation that could lead to this is saving a newly-opened default presentation, like:
prs = Presentation()
output = BytesIO()
prs.save(output)
This of course is not something you would likely do on purpose, but easy enough to do by accident, so I thought I'd mention.
If you can help us repeat your result we'll get it figured out :)

I faced the same problem while developing CGI-like presentation. My pptx shoud be sent as a file to user. You can send pptx as a file using BytesIO, not StringIO
qs = cgi.FieldStorage()
link = qs.getfirst('link', 'default_link_for_debug')
...
...
target_stream = BytesIO()
prs.save(target_stream)
target_stream.seek(0) # important!
length = target_stream.getbuffer().nbytes
buffer = target_stream.getbuffer()
#buffer = target_stream.read() # it also works
if 'HTTP_HOST' in os.environ:
sys.stdout.buffer.write(b'Content-Type: application/vnd.openxmlformats-officedocument.presentationml.presentation\r\n')
sys.stdout.buffer.write('Content-Disposition: attachment; filename="offer-{0}.pptx"\r\n'.format(link).encode('ascii'))
sys.stdout.buffer.write('Content-Length: {0}\r\n'.format(length).encode('ascii'))
sys.stdout.buffer.write(b'Pragma: no-cache\r\n')
sys.stdout.buffer.write(b'\r\n')
sys.stdout.buffer.write(buffer)
else: # for debug
with open("offer-{0}.pptx".format(link),'wb') as out:
out.write(buffer)

Adding new portgroups to vmware virtual switches using pysphere

I'm trying to automate the addition of new portgroups to ESXi hosts using pysphere. I'm using the following code fragment:
from pysphere import MORTypes
from pysphere import VIServer, VIProperty
from pysphere.resources import VimService_services as VI
s = VIServer()
s.connect(vcenter, user, password)
host_system = s.get_hosts().keys()[17]
prop = VIProperty(s, host_system)
propname = prop.configManager._obj.get_element_networkSystem()
vswitch = prop.configManager.networkSystem.networkInfo.vswitch[0]
network_system = VIMor(propname, MORTypes.HostServiceSystem)
def add_port_group(name, vlan_id, vswitch, network_system):
request = VI.AddPortGroupRequestMsg()
_this = request.new__this(network_system)
_this.set_attribute_type(network_system.get_attribute_type())
request.set_element__this(_this)
portgrp = request.new_portgrp()
portgrp.set_element_name(name)
portgrp.set_element_vlanId(vlan_id)
portgrp.set_element_vswitchName(vswitch)
portgrp.set_element_policy(portgrp.new_policy())
request.set_element_portgrp(portgrp)
s._proxy.AddPortGroup(request)
However, when I attempt to run it, I get the following error:
>>> add_port_group(name, vlan_id, vswitch, network_system)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 12, in add_port_group
File "/usr/lib/python2.6/site-packages/pysphere-0.1.8- py2.6.egg/pysphere/resources/VimService_services.py", line 4344, in AddPortGroup
response = self.binding.Receive(AddPortGroupResponseMsg.typecode)
File "/usr/lib/python2.6/site-packages/pysphere-0.1.8- py2.6.egg/pysphere/ZSI/client.py", line 545, in Receive
return _Binding.Receive(self, replytype, **kw)
File "/usr/lib/python2.6/site-packages/pysphere-0.1.8- py2.6.egg/pysphere/ZSI/client.py", line 464, in Receive
raise FaultException(msg)
pysphere.ZSI.FaultException: The object has already been deleted or has not been completely created
I've attempted to swap in different values for "vswitch" and "network_system", but I haven't had any success. Has anyone attempted to do something similar with pysphere successfully?
I can accomplish what I need through Powershell, which demonstrates that it isn't a vmware issue, but I don't want to use Powershell in this particular case.

I tried your code on one of our vSpheres.
It seems you are passing the object to set_element_vswitchName rather than the name. Maybe this will help:
vswitch = prop.configManager.networkSystem.networkInfo.vswitch[0].name