I have a python script which is making a couple of api calls and returning the response to me via email. I want to run this script through a jenkins pipeline job.
I have a token, which I have stored in the jenkins credential manager as a secret text.
The problem is that I am unsure as how to fetch this token in my python script. I have tried looking at a number of solutions, but all of those are leaving me confused.
This is what my jenkins pipeline looks like:
pipeline {
agent {
node {
label 'node1'
}
}
environment {
deva_stross_token=credentials('devadrita-stross') //i have saved the credential with id 'devadrita-stross', and this I understand, is fetching it for my pipeline
}
stages {
stage('running python script') {
steps {
script {
bat """
python -u C://Users//Administrator//Desktop//stross//stross-script.py
"""
}
}
}
}
}
But what changes should I make to fetch it to my script?
Here is the python script.
import requests
import urllib3
import json
import time
import os
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
def initiateScan():
url = ""
payload={}
files=[
('source',('amail.zip',open('C:/Users/Administrator/Desktop/stross/amail.zip','rb'),'application/zip')),
('metadata',('metadata.json',open('C:/Users/Administrator/Desktop/stross/metadata.json','rb'),'application/json'))
]
headers = {
'Authorization': ' Bearer **<token required here>**'
}
response = requests.request("POST", url, headers=headers, data=payload, files=files, verify=False)
resp=response.json()
print(resp)
jobId=resp["job_id"]
return(jobId)
def main():
jobIdFromInitiate=initiateScan()
main()
Thank you in advance for your help!
EDIT-
I understand that there are a few ways to do this, one with credentials() and one with environment variables. The problem is that I am mixing the two up and am not able to implement either method through code. So please help with code.
EDIT 1-
Dividing #rok's answer into steps to follow-
"(1)Add withCredentials block to your Jenkinsfile for reading the credentials and (2)binding them to environment variables. Then, (3)pass the values of the environment variables to your python script as arguments."
pipeline {
agent {
node {
label 'node1'
}
}
environment {
deva_stross_token=credentials('devadrita-stross')
}
withCredentials([string(credentialsId: 'devadrita-stross', variable: 'deva-stross-token')]) {
// some block
}
stages {
stage('saving stross token and printing') {
steps {
script {
bat """
python -u C://Users//Administrator//Desktop//stross//stross-script.py
"""
}
}
}
}
}
(1) added withCredentials block for reading credentials
(2) what is meant by binding the credentials with environment variables? Did the withCredentials block bind the token to environment variable 'deva-stross-token'? Is my environment block useful or needed here?
(3) how to pass values of the environment variables to my python script?
I feel like I should add that I am relatively new to coding, so please bear with me if my questions seem very basic.
Use Credentials Binding plugin. Add withCredentials block to your Jenkinsfile for reading the credentials and binding them to environment variables. Then, pass the values of the environment variables to your python script as arguments.
Jenkins docs list supported credentials types and various bindings.
This is what worked for me:
Jenkinsfile:
stages {
stage('run stross script') {
steps{
withCredentials([string(credentialsId: 'devadrita-stross', variable: 'my_stross_token')]) {
echo "My stross token is '${my_stross_token}'!"
bat "python C:\\Users\\Administrator\\Desktop\\stross\\stross-script.py "
}
}
}
}
This binds my credential to the specified environment variable.
I then import the variable to my python script, and use it like any other variable, i.e.- pass it to functions, etc.
Here is a code snippet:
def main():
token = os.environ['my_stross_token']
print(token)
jobIdFromInitiate=initiateScan(token)
getStatus(jobIdFromInitiate, token)
Related
So I have some Swift code that send a request to my local host
//
// ContentView.swift
// Shared
//
// Created by Ulto4 on 10/23/21.
//
import SwiftUI
struct ContentView: View {
var body: some View {
VStack{
Text("Hello, world!")
.padding()
Button(action : {
self.fu()
}, label: {
Image(systemName: "pencil").resizable().aspectRatio(contentMode:.fit)
})
}
}
func fu(){
let url = URL(string: "http://127.0.0.1:5000/232")
guard let requestUrl = url else { fatalError() }
var request = URLRequest(url: requestUrl)
request.httpMethod = "GET"
let task = URLSession.shared.dataTask(with: request) { (data, response, error) in
if let error = error {
print("Error took place \(error)")
return
}
if let response = response as? HTTPURLResponse {
print("Response HTTP Status code: \(response.statusCode)")
}
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
ContentView()
}
}
}
However, on my Flask app there are no get requests coming in and the function isn't running. There also isn't anything printing to the console.
I am fairly new to swift so I don't really know how to fix this.
Is there any other way to send requests in swift, if not, How would I fix this?
You are creating the URLSessionDataTask, but you never start it. Call task.resume(), e.g.
func performRequest() {
guard let url = URL(string: "http://127.0.0.1:5000/232") else {
fatalError()
}
let task = URLSession.shared.dataTask(with: url) { data, response, error in
if let error = error {
print("Error took place \(error)")
return
}
if let response = response as? HTTPURLResponse {
print("Response HTTP Status code: \(response.statusCode)")
}
}
task.resume() // you must call this to start the task
}
That having been said, a few caveats:
You are doing http rather than https. Make sure to temporarily enable insecure network requests with app transport settings, e.g.
You didn’t say if this was for macOS or iOS.
If running on physical iOS device, it will not find your macOS web server at 127.0.0.1 (i.e., it will not find a web server running on your iPhone). You will want to specify the IP number for your web server on your LAN.
If macOS, make sure to enable outbound network requests in the target’s “capabilities”:
You asked:
Is there any other way to send requests in swift?
It is probably beyond the scope of your question, but longer term, when using SwiftUI, you might consider using Combine, e.g., dataTaskPublisher. When running a simple “what was the status code” routine, the difference is immaterial, but when you get into more complicated scenarios where you have to parse and process the responses, Combine is more consistent with SwiftUI’s declarative patterns.
Let us consider a more complicated example where you need to parse JSON responses. For illustrative purposes, below I am testing with httpbin.org, which echos whatever parameters you send. And I illustrate the use of dataTaskPublisher and how it can be used with functional chaining patterns to get out of the mess of hairy imperative code:
struct SampleObject: Decodable {
let value: String
}
struct HttpBinResponse<T: Decodable>: Decodable {
let args: T
}
class RequestService: ObservableObject {
var request: AnyCancellable?
let decoder = JSONDecoder()
#Published var status: String = "Not started yet"
func startRequest() {
request = createRequest().sink { completion in
print("completed")
} receiveValue: { [weak self] object in
self?.status = "Received " + object.value
}
}
func createRequest() -> AnyPublisher<SampleObject, Error>{
var components = URLComponents(string: "https://httpbin.org/get")
components?.queryItems = [URLQueryItem(name: "value", value: "foo")]
guard let url = components?.url else {
fatalError("Unable to build URL")
}
return URLSession.shared.dataTaskPublisher(for: url)
.map(\.data)
.decode(type: HttpBinResponse<SampleObject>.self, decoder: decoder)
.map(\.args)
.receive(on: DispatchQueue.main)
.eraseToAnyPublisher()
}
}
struct ContentView: View {
#ObservedObject var requestService = RequestService()
var body: some View {
VStack{
Text("Hello, world!")
.padding()
Button {
requestService.startRequest()
} label: {
Image(systemName: "pencil").resizable().aspectRatio(contentMode:.fit)
}
Text(requestService.status)
}
}
}
But, like I said, it is beyond the scope of this question. You might want to make sure you get comfortable with SwiftUI and basic URLSession programming patterns (e.g., making sure you resume any tasks you create). Once you have that mastered, you can come back to Combine to write elegant networking code.
FWIW, like workingdog said, you could also use the new async-await rendition of data(for:delegate:). But when in the declarative world of SwiftUI, I would suggest Combine.
I have aws recognition code written in Python, and it run's by Node API, which works fine on Windows system but when I'm deploying it on Linux I'm facing this issue:- botocore.errorfactory.InvalidS3ObjectException: An error occurred (InvalidS3ObjectException) when calling the DetectText operation: Unable to get object metadata from S3. Check object key, region and/or access permissions.
I have given both AmazonRekognitionFullAccess and AmazonS3ReadOnlyAccess access role to I'm user. Still I don't know how to get things going.
Python code:-
bucket = 'image-test'
def image_to_dict(fileName, bucket):
client = boto3.client('rekognition', 'us-east-2')
response = client.detect_text(Image = { 'S3Object': { 'Bucket': bucket,
'Name': fileName } })
return response
Node Code used to run Python script:-
var options = {
mode: 'text',
pythonPath:"/usr/bin/python2.7"
pythonOptions: ['-u'],
scriptPath: "/home/ubuntu/test",
args: [imageURl]
};
PythonShell.run('script.py', options, function (err, results) {
if (err)
throw err;
console.log("Data is: "+results)
I have Python version 2.7 installed on my Ubuntu, pip version 10.0.1.
Thanks for the help.
The reason behind the issue was, when i was passing image name as argument from Node API, the name was getting manipulated due to some Substring logic.So when python script goes with that manipulated name to search in S3 bucket, it used to through above error as that name was not existing in the S3 bucket.
Is it possible to extract data (to google cloud storage) from a shared dataset (where I have only have view permissions) using the client APIs (python)?
I can do this manually using the web browser, but cannot get it to work using the APIs.
I have created a project (MyProject) and a service account for MyProject to use as credentials when creating the service using the API. This account has view permissions on a shared dataset (MySharedDataset) and write permissions on my google cloud storage bucket. If I attempt to run a job in my own project to extract data from the shared project:
job_data = {
'jobReference': {
'projectId': myProjectId,
'jobId': str(uuid.uuid4())
},
'configuration': {
'extract': {
'sourceTable': {
'projectId': sharedProjectId,
'datasetId': sharedDatasetId,
'tableId': sharedTableId,
},
'destinationUris': [cloud_storage_path],
'destinationFormat': 'AVRO'
}
}
}
I get the error:
googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/sharedProjectId/jobs?alt=json
returned "Value 'myProjectId' in content does not agree with value
sharedProjectId'. This can happen when a value set through a parameter
is inconsistent with a value set in the request.">
Using the sharedProjectId in both the jobReference and sourceTable I get:
googleapiclient.errors.HttpError: https://www.googleapis.com/bigquery/v2/projects/sharedProjectId/jobs?alt=json
returned "Access Denied: Job myJobId: The user myServiceAccountEmail
does not have permission to run a job in project sharedProjectId">
Using myProjectId for both the job immediately comes back with a status of 'DONE' and with no errors, but nothing has been exported. My GCS bucket is empty.
If this is indeed not possible using the API, is there another method/tool that can be used to automate the extraction of data from a shared dataset?
* UPDATE *
This works fine using the API explorer running under my GA login. In my code I use the following method:
service.jobs().insert(projectId=myProjectId, body=job_data).execute()
and removed the jobReference object containing the projectId
job_data = {
'configuration': {
'extract': {
'sourceTable': {
'projectId': sharedProjectId,
'datasetId': sharedDatasetId,
'tableId': sharedTableId,
},
'destinationUris': [cloud_storage_path],
'destinationFormat': 'AVRO'
}
}
}
but this returns the error
Access Denied: Table sharedProjectId:sharedDatasetId.sharedTableId: The user 'serviceAccountEmail' does not have permission to export a table in
dataset sharedProjectId:sharedDatasetId
My service account now is an owner on the shared dataset and has edit permissions on MyProject, where else do permissions need to be set or is it possible to use the python API using my GA login credentials rather than the service account?
* UPDATE *
Finally got it to work. How? Make sure the service account has permissions to view the dataset (and if you don't have access to check this yourself and someone tells you that it does, ask them to double check/send you a screenshot!)
After trying to reproduce the issue, I was running into the parse errors.
I did how ever play around with the API on the Developer Console [2] and it worked.
What I did notice is that the request code below had a different format than the documentation on the website as it has single quotes instead of double quotes.
Here is the code that I ran to get it to work.
{
'configuration': {
'extract': {
'sourceTable': {
'projectId': "sharedProjectID",
'datasetId': "sharedDataSetID",
'tableId': "sharedTableID"
},
'destinationUri': "gs://myBucket/myFile.csv"
}
}
}
HTTP Request
POST https://www.googleapis.com/bigquery/v2/projects/myProjectId/jobs
If you are still running into problems, you can try the you can try the jobs.insert API on the website [2] or try the bq command tool [3].
The following command can do the same thing:
bq extract sharedProjectId:sharedDataSetId.sharedTableId gs://myBucket/myFile.csv
Hope this helps.
[2] https://cloud.google.com/bigquery/docs/reference/v2/jobs/insert
[3] https://cloud.google.com/bigquery/bq-command-line-tool
Make sure the service account has permissions to view the dataset (and if you don't have access to check this yourself and someone tells you that it does, ask them to double check/send you a screenshot!)
When I look at the documentation for passing parameters to the Jasper Report REST 2 API here: http://community.jaspersoft.com/documentation/jasperreports-server-web-services-guide/v550/running-report-asynchronously I see that I need to have a "parameters" dict. The example in the link shows the XML which is not all that useful since it's unclear exactly what the equivalent JSON should look like. The closest I could find is in this link: http://community.jaspersoft.com/documentation/jasperreports-server-web-services-guide/v56/modifying-report-parameters. Now, I am sending the equivalent of that to the server (and every other permutation I can think of), and I continue to get a "400 Client Error: Bad Request" back. I could really use an exact example of the python code to generate the required "parameters" parameter for say "my_parameter_1="test_value_1".
Here is my current POST data (with a few params missing for brevity). I know this is correct since the report works fine if I omit the "parameters" parameter:
{
'outputFormat': 'pdf',
'parameters': [{'name': 'ReportID', 'value': ['my_value_1']}],
'async': 'true',
'pages': '',
'interactive': 'false'
}
Nice Job there Staggart. I got it now. Because I wasn't reading with max. scrutinity, I wasted some additional time. So the interested coder is not only advised to be aware of the nested, syntactictally interesting reportParameter-property, but especially that the value-property inside that is an array. I suppose one could pass some form of Lists/Arrays/Collections here?
What irritated me was, if I should construct more than one "reportParameter" property, but that would be nonsense according to
Does JSON syntax allow duplicate keys in an object.
So just for the record, how to post multiple parameters:
{
"reportUnitUri": "/reports/Top10/Top10Customers",
"async": true,
"freshData": true,
"saveDataSnapshot": false,
"outputFormat": "pdf",
"interactive": false,
"ignorePagination": true,
"parameters": {
"reportParameter": [
{
"name": "DATE_START_STRING",
"value": ["14.07.2014"]
},
{
"name": "DATE_END_STRING",
"value": ["14.10.2014"]
}
]
}
}
If someone accidently is struggling with communicating with jasper via REST and PHP. Do yourself a favour and use the Requests for PHP instead of pure CURL. It even has a fallback for internally using Sockets instead of CURL, when latter isn't available.
Upvote for you Staggart.
OK, thanks to rafkacz1 # http://community.jaspersoft.com/questions/825719/json-equivalent-xml-post-reportexecutions-rest-service who posted an answer, I figured it out. As he report there, the required format is:
"parameters":{
"reportParameter":[
{"name":"my_parameter_1","value":["my_value_1"]}
]
}
Pay particular attention to the plurality of "reportParameter".
Here is an example that worked for me. Im using Python 2.7, and the community edition of Jaspersoft. Like the C# example above, this example also uses the rest v2 which made it very simple for me to download a pdf report quickly
import requests
sess = requests.Session()
auth = ('username', 'password')
res = sess.get(url='http://your.jasper.domain:8080/jasperserver/', auth=auth)
res.raise_for_status()
url = 'http://your.jasper.domain:8080/jasperserver/rest_v2/reports/report_folder/sub_folder/report_name.pdf'
params = {'Month':'2', 'Year':'2017','Project': 'ProjectName'}
res = sess.get(url=url, params=params, stream=True)
res.raise_for_status()
path = '/path/to/Downloads/report_name.pdf'
with open(path, "wb") as f:
f.write(res.content)
Here's a full example about generate a report using Rest V2, in my case it's running on C#:
try {
var server = "http://localhost:8080/jasperserver";
var login = server + "/rest/login";
var report = "/rest_v2/reports/organization/Reports/report_name.pdf";
var client = new WebClient();
//Set the content type of the request
client.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
//Set the username and password
NameValueCollection parametros = new NameValueCollection();
parametros.Add("j_username", "jasperadmin");
parametros.Add("j_password", "123456");
//Request to login
client.UploadValues(login, "POST", parametros);
//Get session cookie
string session = client.ResponseHeaders.Get("Set-Cookie");
//Set session cookie to the next request
client.Headers.Add("Cookie", session);
//Generate report with parameters: "start" and "end"
var reporte = client.DownloadData(server + report + "?start=2015-10-01&end=2015-10-10");
//Returns the report as response
return File(reporte, "application/pdf", "test.pdf");
}catch(WebException e){
//return Content("There was a problem, status code: " + ((HttpWebResponse)e.Response).StatusCode);
return null;
}
I have tried Qooxdoo and I made a simple Python server with SimpleXMLRPCServer, with a Python test I get the data without problems, but can I get this data from Qooxdoo? I get lost, and I've searched for 3 days but didn't get solutions.
I try this:
var JSON_lista_empresas = 1000
button1.addListener("execute", function(e)
{
var rpc = new qx.io.remote.Rpc();
rpc.setServiceName("get_data");
//rpc.setCrossDomain(true);
rpc.setUrl("http://192.168.1.54:46000");
rpc.addListener("completed", function(event)
{
console.log(event.getData());
});
rpc.callAsync( JSON_lista_empresas, '');
});
And I tried other options but got nothing :(
The link to files:
http://mieresdelcamin.es/owncloud/public.php?service=files&dir=%2Fjesus%2Ffiles%2FQooxdoo
I tried and read all of qooxdoo-contrib.
Well,
RpcPython --> Ok
and in class/qooxdoo -> test.py
run server [start-server.py] and query from webroser:
http://127.0.0.1:8000//?_ScriptTransport_id=1&nocache=1366909868006&_ScriptTransport_data={%22service%22%3A%22qooxdoo.test%22%2C%22method%22%3A%22echo%22%2C%22id%22%3A1%2C%22params%22%3A[%22Por%20fin%22]}
and the reply in webroser is:
qx.io.remote.ScriptTransport._requestFinished(1,{"error": null, "id": 1, "result": "Client said: [ Por fin ]"});
but if i query from qooxdoo like the reply is [error.png]
The code for qooxdoo:
var rpc = new qx.io.remote.Rpc( "http://127.0.0.1:8000/");
rpc.setCrossDomain( true);
rpc.setServiceName( 'qooxdoo.test');
// asynchronous call
var handler = function(result, exc) {
if (exc == null) {
alert("Result of async call: " + result);
} else {
alert("Exception during async call: " + exc+ result);
}
};
rpc.callAsync(handler, "echo", "Por fin");
I lost :((
Files in:
http://mieresdelcamin.es/owncloud/public.php?service=files&dir=%2Fjesus%2Ffiles%2FQooxdoo
Well, with Firebug this error in owncloud qx.io.remote.ScriptTransport.....is detect
¿?.............
Best Regards.
I'm guessing you confuse XML-RPC with JSON-RPC and qooxdoo only supports the latter. These protocols are similar but the data interchange format is different (XML or JSON). Instead of the SimpleXMLRPCServer you could use "RpcPython" on the server side which is a qooxdoo contrib project.
See:
http://qooxdoo.org/contrib/project/rpcpython
http://sourceforge.net/p/qooxdoo-contrib/code/HEAD/tree/trunk/qooxdoo-contrib/RpcPython/
Once you have this server up and running you should be able to test it:
http://manual.qooxdoo.org/2.1.1/pages/communication/rpc_server_writer_guide.html#testing-a-new-server
http://sourceforge.net/p/qooxdoo-contrib/code/HEAD/tree/trunk/qooxdoo-contrib/RpcPython/trunk/services/class/qooxdoo/test.py
After that your qooxdoo (client) code hopefully works also. :)
Ok,
In the file http.py of qxjsonrc module in the line 66 change
response='qx.io.remote.ScriptTransport._requestFinished(%s,%s);'%(scriptTransportID,response)
for
response='qx.io.remote.transport.Script._requestFinished(%s,%s);'%(scriptTransportID,response)
And run fine :))
This link for package modified:
http://mieresdelcamin.es/owncloud/public.php?service=files&dir=%2Fjesus%2Ffiles%2FQooxdoo
Best Regards and thanks!!!
As Richard already pointed Qooxdoo only supports its flavor of JSON-RPC.
I maintain a fork of original rpcpython called QooxdooCherrypyJsonRpc. The main goal was to hand over transport protocol to some robust framework, and leave only JSON RPC stuff. CherryPy, obviously a robust framework, allows HTTP, WSGI and FastCGI deployment. Code was refactored and covered with tests. Later I added upload/download support and consistent timezone datetime interchange.
At very minimum your Python backend may look like (call it test.py):
import cherrypy
import qxcpjsonrpc as rpc
class Test(rpc.Service):
#rpc.public
def add(self, x, y):
return x + y
config = {
'/service' : {
'tools.jsonrpc.on' : True
},
'/resource' : {
'tools.staticdir.on' : True,
'tools.staticdir.dir' : '/path/to/your/built/qooxdoo/app'
}
}
cherrypy.tools.jsonrpc = rpc.ServerTool()
if __name__ == '__main__':
cherrypy.quickstart(config = config)
Then you can do in your qooxdoo code as follows:
var rpc = new qx.io.remote.Rpc();
rpc.setServiceName('test.Test');
rpc.setUrl('http://127.0.0.1:8080/service');
rpc.setCrossDomain(true); // you need this for opening app from file://
rpc.addListener("completed", function(event)
{
console.log(event.getData());
});
rpc.callAsyncListeners(this, 'add', 5, 7);
Or open the link directly:
http://127.0.0.1:8080/service?_ScriptTransport_id=1&_ScriptTransport_data=%7B%22params%22%3A+%5B12%2C+13%5D%2C+%22id%22%3A+1%2C+%22service%22%3A+%22test.Test%22%2C+%22method%22%3A+%22add%22%7D
For more info take a look at the package page I posted above.
Richard Sternagel wrote about rpcpython. This version of rpcpython doesn't work with present version of simplejson. Becouse in json.py have incorrect import:
from simplejson.decoder import ANYTHING
from simplejson.scanner import Scanner, pattern
Improve rpcpython or use another server, for example CherryPy.