I'm having some problems trying to do an ItemSearch on the Blended index using the Amazon Product API.
According to the documentation, Blended requests cannot specify the MerchantId parameter - and indeed, if I try to include it I get an error telling me so. However, when I don't include it, I get an error telling me that my request is missing a required parameter combination and that a valid combination includes MerchantId... what the hell?
The failing requests are being sent as part of batches with other requests that are succeeding. I'm using REST to send my requests, so here's an example:
http://ecs.amazonaws.com/onca/xml?AWSAccessKeyId=-------------&
ItemSearch.1.Keywords=Mates%20of%20State&
ItemSearch.1.MerchantId=Amazon&
ItemSearch.1.SearchIndex=DVD&
ItemSearch.2.Keywords=teaching%20Lily%20various%20computer%20related%20skills&
ItemSearch.2.SearchIndex=Blended&
ItemSearch.Shared.Availability=Available&
ItemSearch.Shared.Condition=All&
ItemSearch.Shared.ResponseGroup=Small%2CSalesRank%2CImages%2COfferSummary%2CSimilarities&
Operation=ItemSearch%2CSimilarityLookup&
Service=AWSECommerceService&
SimilarityLookup.1.ItemId=B000FNNHZ2&
SimilarityLookup.2.ItemId=B000EQ5UPU&
SimilarityLookup.Shared.Availability=Available&
SimilarityLookup.Shared.Condition=All&
SimilarityLookup.Shared.MerchantId=Amazon&
SimilarityLookup.Shared.ResponseGroup=Small%2CSalesRank%2CImages%2COfferSummary&
Timestamp=2010-04-02T17%3A18%3A05Z&
Signature=----------------
Here's the XML response:
<Items xmlns="http://webservices.amazon.com/AWSECommerceService/2005-10-05">
<Request>
<IsValid>False</IsValid>
<ItemSearchRequest>
<Availability>Available</Availability>
<Condition>All</Condition>
<Keywords>teaching Lily various computer related skills</Keywords>
<ResponseGroup>Similarities</ResponseGroup>
<ResponseGroup>SalesRank</ResponseGroup>
<ResponseGroup>OfferSummary</ResponseGroup>
<ResponseGroup>Small</ResponseGroup>
<ResponseGroup>Images</ResponseGroup>
<SearchIndex>Blended</SearchIndex>
</ItemSearchRequest>
<Errors>
<Error>
<Code>AWS.MissingParameterCombination</Code>
<Message>Your request is missing a required parameter combination. Required parameter combinations include MerchantId, Availability.</Message>
</Error>
</Errors>
</Request>
</Items>
Any ideas as to what I'm doing wrong?
I seem to have solved this by removing both the Availability and Condition parameters. I'd ideally prefer to be able to filter by availability, but at least it's working.
Related
I would like to retrieve some historical stock prices via a REST API from the following site:
https://www.boerse-frankfurt.de/zertifikat/de0007873291-open-end-zertifikat-auf-dow-jones-industrial-average
The response is a JSON.
Basically, the query can be done as follows: An OPTIONS call is sent without parameters and then a GET request with header parameters.
Both calls are sent to the following address:
https://api.boerse-frankfurt.de/v1/data/quote_history_derivatives?isin=DE0007873291&mic=XSC&from=2021-11-12T07%3A00%3A00.000Z&to=2021-11-12T21%3A00%3A00.000Z&offset=0&limit=25
The following two parameters are included in the header:
Client-Date: 2021-11-16T23:02:29.529Z
X-Client-TraceId: d2d6911d81ebbbff7a7549555a2c26d6
And now my question: how do you get the X-Client-TraceId? It looks like a UUID, but it doesn't seem to be one. The value changes with every page view in the browser. But you can't just enter any value.
Many greetings,
Trebor
Since this question was asked, someone has written a blog post about this exact topic. The algorithm detailed there still seems to be in use (as of 2022-03-12).
An excerpt of the relevant parts:
Client-Date
This is the current time, converted to a string with Javascript’s toISOString() function.
[...]
X-Client-TraceId
[...]
salt is a fixed string, in this case w4icATTGtnjAZMbkL3kJwxMfEAKDa3MN. Apparently it appears in the source code as-is so it must be constant.
X-Client-TraceId is the md5 of time + url + salt.
Note: time is the string sent in the Client-Date header.
The blog post has some additional information around the process of reverse engineering this algorithm and the X-Security header.
I'm making a Community Connector with the following fields, among others: Age, gender and impressions.
When I try to do a bar chart with Impressions as a metric, Age as a dimension and Gender as a breakdown dimension (or Gender and Age, inverted) I get the following error:
User Configuration Error
This data source was improperly configured.
Invalid argument type.
Error ID: b44d6288
Debugging, I found the problem is that it isn't making a single request to getData() including the three fields (which, when processed, would make the right call to the API and get the right data). Instead it only requests the pair dimension-metric on one request and sometimes also breakdown dimension-metric on others (and, sometimes, dimension-metric with filter info), which gives it a "broken" data which it apparently can't make sense of. As the request to my getData() only includes two fields, I return two fields per row. Info about the third one can't be found nowhere, especially on the request parameter, as far as I can see.
This behavior appeared somewhere along the development of the connector -- at some points this exact combination worked normal.
As this behavior doesn't include code it's really scaring me. Any idea would be deeply appreciated.
(Possibly a duplicate of Can't send a keyedMessage to brokers with partitioner.class=kafka.producer.DefaultPartitioner, although the OP of that question didn't mention kafka-python. And anyway, it never got an answer.)
I have a Python program that has been successfully (for many months) sending messages to the Kafka broker, using essentially the following logic:
producer = kafka.KafkaProducer(bootstrap_servers=[some_addr],
retries=3)
...
msg = json.dumps(some_message)
res = producer.send(some_topic, value=msg)
Recently, I tried to upgrade it to send messages to different partitions based on a definite key value extracted from the message:
producer = kafka.KafkaProducer(bootstrap_servers=[some_addr],
key_serializer=str.encode,
retries=3)
...
try:
key = some_message[0]
except:
key = None
msg = json.dumps(some_message)
res = producer.send(some_topic, value=msg, key=key)
However, with this code, no messages ever make it out of the program to the broker. I've verified that the key value extracted from some_message is always a valid string. Presumably I don't need to define my own partitioner, since, according to the documentation:
The default partitioner implementation hashes each non-None key using the same murmur2 algorithm as the java client so that messages with the same key are assigned to the same partition.
Furthermore, with the new code, when I try to determine what happened to my send by calling res.get (to obtain a kafka.FutureRecordMetadata), that call throws a TypeError exception with the message descriptor 'encode' requires a 'str' object but received a 'unicode'.
(As a side question, I'm not exactly sure what I'd do with the FutureRecordMetadata if I were actually able to get it. Based on the kafka-python source code, I assume I'd want to call either its succeeded or its failed method, but the documentation is silent on the point. The documentation does say that the return value of send "resolves to" RecordMetadata, but I haven't been able to figure out, from either the documentation or the code, what "resolves to" means in this context.)
Anyway: I can't be the only person using kafka-python 1.3.3 who's ever tried to send messages with a partitioning key, and I have not seen anything on teh Intertubes describing a similar problem (except for the SO question I referenced at the top of this post).
I'm certainly willing to believe that I'm doing something wrong, but I have no idea what that might be. Is there some additional parameter I need to supply to the KafkaProducer constructor?
The fundamental problem turned out to be that my key value was a unicode, even though I was quite convinced that it was a str. Hence the selection of str.encode for my key_serializer was inappropriate, and was what led to the exception from res.get. Omitting the key_serializer and calling key.encode('utf-8') was enough to get my messages published, and partitioned as expected.
A large contributor to the obscurity of this problem (for me) was that the kafka-python 1.3.3 documentation does not go into any detail on what a FutureRecordMetadata really is, nor what one should expect in the way of exceptions its get method can raise. The sole usage example in the documentation:
# Asynchronous by default
future = producer.send('my-topic', b'raw_bytes')
# Block for 'synchronous' sends
try:
record_metadata = future.get(timeout=10)
except KafkaError:
# Decide what to do if produce request failed...
log.exception()
pass
suggests that the only kind of exception it will raise is KafkaError, which is not true. In fact, get can and will (re-)raise any exception that the asynchronous publishing mechanism encountered in trying to get the message out the door.
I also faced the same error. Once I added json.dumps while sending the key, it worked.
producer.send(topic="first_topic", key=json.dumps(key)
.encode('utf-8'), value=json.dumps(msg)
.encode('utf-8'))
.add_callback(on_send_success).add_errback(on_send_error)
I'm currently trying to configure the Fiware Iot Broker with the Configuration Manager (NECongMan) and Fiware Orion as the context producer. I'm having a problem with the NGSI10 subscribeContext operation.
This is the request sent to the IoT Broker:
<?xml version="1.0"?>
<subscribeContextRequest>
<entityIdList>
<entityId type="Room" isPattern="false">
<id>Room1</id>
</entityId>
</entityIdList>
<attributeList>
<attribute>temperature</attribute>
</attributeList>
<reference>http://localhost:1028/accumulate</reference>
<duration>PT1H</duration>
<notifyConditions>
<notifyCondition>
<type>ONCHANGE</type>
<condValueList>
<condValue>pressure</condValue>
</condValueList>
</notifyCondition>
</notifyConditions>
<throttling>PT5S</throttling>
</subscribeContextRequest>
The IoTBroker tries to contact Orion issuing a subscribeContextRequest but the duration string is changed:
<subscribeContextRequest>
<entityIdList>
<entityId
type="Room"
isPattern="false">
<id>
Room1
</id>
</entityId>
</entityIdList>
<attributeList>
<attribute>
temperature
</attribute>
</attributeList>
<reference>
http://192.168.16.178:8080/ngsi10/notify
</reference>
<duration>
P0Y0M0DT0H59M58.157S
</duration>
<notifyConditions>
<notifyCondition>
<type>
ONCHANGE
</type>
<condValueList>
<condValue>
pressure
</condValue>
</condValueList>
</notifyCondition>
</notifyConditions>
<throttling>
PT5S
</throttling>
</subscribeContextRequest>
But Orion gives an error (invalid payload: syntax error in duration string). Do you have any idea how to resolve the issue?
For one hour, Duration parameter may be P1H instead of PT1H, according to the ISO 8601 standard format, it seems like PT is used only to avoid ambiguity between P1M for one month and PT1M for one minute.
Hope it helps
Orion 0.26.1 (the last version in the moment of writing this) doesn't support decimal values (as 58.157) for seconds in duration strings. We have created an issue about that and our plan is to have it solved by next release (0.27.0).
Not sure if modifying the way IoTBroker behaves (in order to round up/down the secons to an integer in the request it sends to Orion) would be a valid workaround... I don't know the details about IoTBroker.
EDIT: the fix to support decimal fractions in the seconds field has been just implemented in Orion in develop branch, so it will be available in the next Orion version (0.27.0). Alternatively, you could download source code and build the code to get the fix right now.
I have a collection of IDs of RESTful resources (all the same type of resource), the number of which can be indefinitely large. I want to make a REST call to get the names of these resources. Something like this:
Send:
['005fc983-fe41-43b5-8555-d9a2310719cd', '4c6e6898-e519-4bac-b03e-e8873d3fa3f0',...]
Receive:
['Resource A', 'Resource B',...]
What is the best way to retrieve the names of these resources RESTfully?
Here are the ideas I have had and the problems I see with each approach:
The naive approach is to iterate through all IDs in my collection and do a 'GET /resource/:id' for each ID. This would be prohibitively slow and resource intensive because of the large number of HTTP calls I would have to make.
The next approach I thought of is to pass the IDs as parameters to a single GET call. The problem here is that most servers have a limit on the URL length, which would be quickly exceeded.
Next, I thought that putting the IDs in the body of a GET would work, but according to Roy Fielding, data in the GET body should not affect the results of a REST call: HTTP GET with request body
I could use a POST request and put the data on the POST body, but POST is intended for creating and modifying resources, which is not what I'm doing. Maybe I should ignore the intent of the verb and use it anyway?
I could split the request into multiple GET requests to avoid exceeding the max URL length. The problem here is that I have to combine the results after all calls have returned, which is potentially slow.
I could create a collection resource within my main resource by posting my list of IDs to 'POST /resource/collection', then use a 'GET /resource/collection/:id' call to retrieve the results. This actually works, but then I have to do a 'DELETE /resource/collection/:id' to clean up. It takes multiple calls, requires cleanup, and seems a bit clunky overall, so it's okay, but not ideal.
Is there a better way to do this?
Your last approach is RESTful and the one I recommend. I'd do this:
Step 1:
Request:
POST /resource/collection
Content-Tpye: application/json
{
"ids": [
"005fc983-fe41-43b5-8555-d9a2310719cd",
"4c6e6898-e519-4bac-b03e-e8873d3fa3f0"
]
}
Response:
201 Created
Location: /resource/collection/89AB8902-FDF1-11E4-ADDF-CD4FB664A5DC
Step 2:
Request:
GET /resource/collection/89AB8902-FDF1-11E4-ADDF-CD4FB664A5DC
Response:
200 OK
Content-Type: application/json
{
"resources": [ ... ]
}
but then I have to do a 'DELETE /resource/collection/:id' to clean up.
Not, that is not necessary. The server could implement a job that removes all collections that are older than a specific timestamp. It is not the client who has to do this.
If later a client access the collection again, the server would respond with
410 Gone