I'm going crazy trying to get data through an API call using request and pandas. It looks like it's nested data, but I cant get the data i need.
https://xorosoft.docs.apiary.io/#reference/sales-orders/get-sales-orders
above is the api documentation. I'm just trying to keep it simple and get the itemnumber and qtyremainingtoship, but i cant even figure out how to access the nested data. I'm trying to use DataFrame to get it, but am just lost. any help would be appreciated. i keep getting stuck at the 'Data' level.
type(json['Data'])
df = pd.DataFrame(['Data'])
df.explode('SoEstimateHeader')
df.explode('SoEstimateHeader')
Cell In [64], line 1
df.explode([0:])
^
SyntaxError: invalid syntax
I used the link to grab a sample response from the API documentation page you provided. From the code you provided it looks like you are already able to get the data and I'm assuming the you have it as a dictionary type already.
From what I can tell I don't think you should be using pandas, unless its some downstream requirement in the task you are doing. But to get the ItemNumber & QtyRemainingToShip you can use the code below.
# get the interesting part of the data out of the api response
data_list = json['Data']
#the data_list is only one element long, so grab the first element which is of type dictionary
data = data_list[0]
# the dictionary has two keys at the top level
so_estimate_header = data['SoEstimateHeader']
# similar to the data list the value associated with "SoEstimateItemLineArr" is of type list and has 1 element in it, so we grab the first & only element.
so_estimate_item_line_arr = data['SoEstimateItemLineArr'][0]
# now we can grab the pieces of information we're interested in out of the dictionary
qtyremainingtoship = so_estimate_item_line_arr["QtyRemainingToShip"]
itemnumber = so_estimate_item_line_arr["ItemNumber"]
print("QtyRemainingToShip: ", qtyremainingtoship)
print("ItemNumber: ", itemnumber)
Output
QtyRemainingToShip: 1
ItemNumber: BC
Side Note
As a side note I wouldn't name any variables json because thats also the name of a popular library in python for parsing json, so that will be confusing to future readers and will clash with the name if you end up having to import the json library.
I have written a script on publish channel and would be triggered before processing. I have to fetch the GLaccount and when trying to do that it returns null value. The piece of code is
irData.getCurrentData("GLDEBITACCT) or even this irData.getCurrentData("VALUE"). I guess this is incorrect because in the XML the GL account appears something like below,
<GLDEBITACCT>
<VALUE>00.00000.00000.000000</VALUE>
<GLCOMP glorder="0">00</GLCOMP><GLCOMP glorder="1">00000</GLCOMP>
<GLCOMP glorder="2">00000</GLCOMP><GLCOMP glorder="3">000000</GLCOMP>
</GLDEBITACCT>
There is a getGL method in psdi.iface.mic.StructureObject . If you pass the GL component name it will fetch value.
I am having quite a bit of trouble with getting the correct form data saved to a server via POST with Requests (2.8.1) module.
I have previous code which does exactly what I want it to do: it encodes a bunch of key:value pairs into the correct header:value payload dict format, and successfully POSTS to the URI. I get a 200 response (what I'm looking for) and everything is great.
This is a section of the OLD payload encoding function, with a ton of key:value pairs omitted for brevity.
Note: the checkbox value set could be any sequence of numbers between 1 and 25, I just wrote it as
item in range(1,5)
to illustrate that the list is comprised of int numbers, i.e. [ "", 1, 2, 3, 4, 5,...] or [ "", 2, 7, 5, 1, 25,...] etc.
checkboxList = ["",]
for item in range(1,5):
checkboxList.append(item)
payload['checkbox[ids][]'] = checkboxList
...
response = request.post(data_url, data=payload)
>> 200 OK!
Here is a print of what the payload dict (checkboxes) looks like before it's sent to the server:
{... "checkbox[ids][]" : [ "", 2, 17, 20, 5], ...}
And when I look on the page with a browser, all the payload information has been correctly recorded (omitted above) AND the checkboxes (shown above) are correct!
Originally, the checkbox values came from an excel file, as did the rest of the information that was put into the payload before being POSTed to the server. However, now I'm retrieving the information from an SQLite db.
Below is the NEW code that records the checkboxes incorrectly. I should note: I do not have access to the server, so I cannot easily tell if it's a server issue, but let's assume it's not the servers fault. I've had this issue previously, but I got it to work with the above code. However, now that I've started to store the values I need in a db, I cannot get the correct checkboxes recorded by the server.
This is what the data from the db column looks like:
12-5-1-22-4
(... I know this isn't great practice for DB mgmt, but I assume this isn't why the POST is recording the wrong data, and I wanted this question to be as closely representational to my code as possible.)
checkList = checkboxesFromDB.split('-')
payload['checkbox[ids][]'] = checkList
...
response = request.post(data_url, data=payload)
>> 200 OK!
When I look at the site with the browser, it records the checkboxes incorrectly. Now, i should note that 3 checkboxes are selected no matter what I pass to payload[checkbox[ids][]]
It's ALWAYS the same 3, incorrect checkboxes, even if I completely omit checkbox[ids][] from the payload dict. Knowing that, we could assume its a server issue. However, the nearly EXACT code from above works (when I grab the info from an excel file).
I've tried the following (with only one value as a test) without getting the correct checkboxes recorded by the server:
payload['checkbox[ids][]'] = '1'
payload['checkbox[ids][]'] = 1
payload['checkbox[ids][]'] = [1]
payload['checkbox[ids][]'] = ["",1]
payload['checkbox[ids][]'] = [1,""]
When uploading images to the same server, I had an encoding issue when retrieving the image BLOB from the db and trying to pass the buffer object directly to Requests as a file, but I fixed this with cStringIO encoding. (It took me forever as I'm really new to programming, and still unsure of syntax, let alone ways to handle this sort of stuff....) I thought I might be having a similar encoding issue, but with the testing and research I've done, I cannot determine either way as I feel like I'm a bit over my head.
I apologize if this is completely NOOB, but I've done extensive research, trying so many different things that I could think of. I tried passing strings, lists, dicts, forcing encoding of lists as utf-8.
The main reason I'm so perplexed is my original code WORKS, and my new code is nearly identical but doesn't. The only real difference I can think of is now my information is coming from a SQLite db (this particular checkbox column is TEXT type)
Can anyone help me, or point me in a new direction I haven't thought of/know of?
I went through all payload pairs to find that it was an issue with HTML.
I was saving HTML in my SQLite db (via BeautifulSoup without prettifying it) as TEXT. Then I was retrieving it and sending it as a string. This was throwing off the server response.
I have since swapped that sql column value type to VARCHAR (as is best for my use) and prettify it like this foo = bar.prettify(formatter="html")before saving to the db. Now, when i retrieve the value and pass it to the payload, everything works as it should.
I'm running QuickFix with the Python API and connecting to a TT FIX Adapter using FIX4.2
I am logging on and sending a market data request for two instruments. That works fine and data from the instruments comes in as expected. I can get all kinds of information from the messages.
However, I am having trouble getting the Symbol (flag 55) field.
import quickfix as fix
def fromApp(self, message, sessionID):
ID = fix.Symbol()
message.getField(ID)
print ID
This works for the very first message [the initial Market Data Snapshot (flag 35 = W)] that comes to me. Once I start getting incremental refreshes (flag 35 = X), I can no longer get the Symbol field. Every message that arrives results in a Field Not Found error.
This is confusing me because in the logs, the Symbol field is always present, whether the message type is W or X.
Thinking the Symbol is in the header of refresh messages, I tried get.Field(ID) when 35 = W and get.Header().getField(ID) when 35 = X, however this did not work.
Can somebody help me figure out what is going on here? I would like to be able to explicitly tell my computer what instruments it is looking at.
Thanks
Your question is pretty simple, but you've mixed in some misconceptions as well.
1) Symbol will never be in the header. It is a body field.
2) In X messages, the symbol is in a repeating group. You first have to get a group object with msg.GetGroup(), then get the symbol from that. See this example code, from the repeating groups doc page.
3) In W messages, the symbol is not in a group. That's why it works for you there.
It seems clear you are pretty new to QuickFIX and FIX in general. I think you should take few minutes and skim through the "Working with Messages" section of the docs.
Also, the FIXimate website can be your best friend.
I'm missing something obvious here. I am trying to process a POST request that contains a mixture of single value and multi value variables. I can get the single valued variables using request.POST.get('variable_name'), for example:
logging.debug('sale_date: ' + request.POST.get('SALEDATE'))
However, I can't get the multi value variables using request.POST.getlist('variable_name'). For example, the following returns an empty list.
prices = request.POST.getlist("IPN_PRICE")
I can't show all the fields in the request here, because it's work for a client. However this log call:
logging.debug(repr(request.POST))
gives this output (start only)
<QueryDict: {u'IPN_PRICE[]': [u'15.76'], ...
By the way, the request I'm trying to process is an IPN (Instant Payment Notification) from a payment processing service.
prices = request.POST.getlist("IPN_PRICE[]")
This should do the trick.