I am trying to listen and consume an EMS queue message and transform it to insert into a Database using Anypoint Studio platform.
Below youn can find an example of the EMS queue message, and the structure of the target database table, as well as, the error I am getting, I cannot find anything to help me with this and would appreciate any insight on how to solve it! Thank you!
EMS Queue message: 1 Lopez, Gerardo 22/07/1994 323 Corona, San Pedro, Mexico 8177228822
Target DB Table Columns: id, first_name, last_name, birth_date, street, city, country, phone
I need to insert the appropriate data in the corresponding column and transform all characters to lower case, below you can find what I am doing right not and the error I get!
Caused by: org.mule.weave.v2.module.reader.ReaderParsingException: Unexpected character 'G' at payload#[1:4] (line:column), expected Expecting end of input but got G, while reading payload` as Json.
The same message repeats for all EMS Queue messages!
I guess that EMS means TIBCO EMS. It looks that the application is receiving a message from the TIBCO EMS queue, which is the string:
16 Gonzalez, Martin...
For some reason, DataWeave is trying to parse it like a JSON document. That is clear in the error. It fails in the G because the beginning is a number, so the parser for JSON wasn't expecting a G character there. A number by itself would be a perfectly valid JSON document.
You'll have to set the correct format for the input. If the TIBCO EMS connector is receiving the application/json from the message, it should be changed from the sending side.
If you need to force the format you can use set payload though I don't recommend to force it unless there is no other choice:
<set-payload value="#[payload]" mimeType="text/plain" >
Related
I'm making a Community Connector with the following fields, among others: Age, gender and impressions.
When I try to do a bar chart with Impressions as a metric, Age as a dimension and Gender as a breakdown dimension (or Gender and Age, inverted) I get the following error:
User Configuration Error
This data source was improperly configured.
Invalid argument type.
Error ID: b44d6288
Debugging, I found the problem is that it isn't making a single request to getData() including the three fields (which, when processed, would make the right call to the API and get the right data). Instead it only requests the pair dimension-metric on one request and sometimes also breakdown dimension-metric on others (and, sometimes, dimension-metric with filter info), which gives it a "broken" data which it apparently can't make sense of. As the request to my getData() only includes two fields, I return two fields per row. Info about the third one can't be found nowhere, especially on the request parameter, as far as I can see.
This behavior appeared somewhere along the development of the connector -- at some points this exact combination worked normal.
As this behavior doesn't include code it's really scaring me. Any idea would be deeply appreciated.
Is there a way to determine the max value of an argument in a requests.post command to a website if we don't know the amount of data in the website's dataset? I'm trying to execute the following code to get specific information on all daycares from this website, but don't know the value of the last argument (length). Currently, I'm assuming this value is 20, but it is subject to change from time to time. How do I keep it open ended so I don't have to guess the max value for lenth? Code as follows:
data_requested = requests.post("https://data.nj.gov/views/INLINE/rows.json?"
"accessType=WEBSITE&method=getByIds&asHashes=true&start=0&length=20",
json=data)
njcc_data = data_requested.json()
Notice that this has nothing to do with requests.post - the range of values length can take is determined by the creator of that API and is an unknown quantity both to you and to requests.
You can try to reason about what possible values it could take, is it the length of a person? If yes, it's probably not going to be more than 250cm.
You can also use trial and error and see how high you can make it before the API endpoint gives back an error, but I guess this is what you were trying to avoid.
If length is the number of items returned (the length of the returned json array) then you could just try setting it to a high number like 1000 and see if you can get away with it.
I am working on migration of 3.0 code into new 4.2 framework. I am facing a few difficulties:
How to do CDR level deduplication in new 4.2 framework? (Note: Table deduplication is already done).
Where to implement PostDedupProcessor - context or chainsink custom? In either case, do I need to remove duplicate hashcodes from the list or just reject the tuples? Here I am also doing column updating for a few tuples.
My file is not moving into archive. The temporary output file is getting generated and that too empty and outside load directory. What could be the possible reasons? - I have thoroughly checked config parameters and after putting logs, it seems correct output is being sent from transformer custom, so I don't know where it is stuck. I had printed TableRowGenerator stream for logs(end of DataProcessor).
1. and 2.:
You need to select the type of deduplication. It is not a big difference if you choose "table-" or "cdr-level-deduplication".
The ite.businessLogic.transformation.outputType does affect this. There is one Dedup only. You can not have both.
Select recordStream for "cdr-level-deduplication", do the transformation to table row format (e.g. if you like to use the TableFileWriter) in xxx.chainsink.custom::PostContextDataProcessor.
In xxx.chainsink.custom::PostContextDataProcessor you need to add custom code for duplicate-handling: reject (discard) tuples or set special column values or write them to different target tables.
3.:
Possibly reasons could be:
Missing forwarding of window punctuations or statistic tuple
error in BloomFilter configuration, you would see it easily because PE is down and error log gives hints about wrong sha2 functions be used
To troubleshoot your ITE application, I recommend to enable the following debug sinks if checking the StreamsStudio live graph is not sufficient:
ite.businessLogic.transformation.debug=on
ite.businessLogic.group.debug=on
ite.businessLogic.sink.debug=on
Run a test with a single input file only and check the flow of your record and statistic tuples. "Debug sinks" write punctuations markers also to debug files.
File "/home/malikarumi/Projects/cannon/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 2390, in get_db_prep_value
value = uuid.UUID(value)
File "/usr/lib/python2.7/uuid.py", line 134, in __init__
raise ValueError('badly formed hexadecimal UUID string')
ValueError: Problem installing fixture '/home/malikarumi/Projects/cannon/jamf/essell/fixtures/test22byhand.json': badly formed hexadecimal UUID string
I've found the following links so far:
https://github.com/dcramer/django-uuidfield/issues/40
https://github.com/dcramer/django-uuidfield/commit/caae1bc4e45445a06dd11bb22da6a9f07395f78a
Django UUIDField modelfield causes error in Django admin: badly formed hexadecimal UUID string
Django Primary Key: badly formed hexadecimal UUID string
I counted my uuidfield value. It is len=36, because it has dashes in it. At least the string representation I can see is that way. So I replaced it with the same alphanumeric without dashes, as suggested as a test by the bugfix, but I still got the same result.
I checked the model, but there is no max length on any uuid field, nor on the fk link back to the uuid. There's nothing on the fk to suggest it is, or should be limited to, chars, ints, uuids, etc.
Then I found this: http://arthurpemberton.com/2015/04/fixing-uuid-is-not-json-serializable which I hacked into /python2.7/site-packages/django/core/serializers/python.py. The blogger had put it into models.py. But I got the same error, before realizing it was NOT coming from serializers/python.py, as it was yesterday, but from /usr/lib/python2.7/uuid.py, line 134, in init. the relevant portions of that code are:
if hex is not None:
hex = hex.replace('urn:', '').replace('uuid:', '')
hex = hex.strip('{}').replace('-', '')
if len(hex) != 32:
raise ValueError('badly formed hexadecimal UUID string')
int = long(hex, 16)
Rather than try to hack more core code, given that the indication is the problem is json, not Python, I left this alone for now.
Finally, I looked at this:
https://code.djangoproject.com/ticket/24012
It is stated a couple of times here that Django's "UUIDField generates UUIDs in Python". Now here is some history. I created one row, a single instance of Model A into Django with a fixture that had no uuid and no datefield and had no issues. (The uuidfield is on an abstract model, so it is created when the object is created). I did that because I needed the uuid of that Model A instance for a fk field in Model B, which is the one I am struggling with now. I did that by copy pasting the Model A uuid into the fk field on Model B in a csv file which I then converted to json in order to use it as a fixture.
Is it possible that the uuid ran into problems in this copy paste maneuver, before the conversion to json?
If not, that means even though it was an acceptable Python object when it was created, going thru the json conversion messed it up, correct?
If that's the case, what is a workaround?
Can the Arthur Pemberton code be made to work somewhere else in this process?
If I leave the uuid off, I can probably make this work, but then I have to go back and put the all the fk uuid's in manually. Is there a better solution? Maybe a bulk insert of that field alone?
This may be a recurring issue for me, because I am also using Scrapy, which supports but does not require json. None of my scraped items will come with uuid, but how do I automate adding their fk's into my process in order to get them into Django?
Or is all of this a good reason to forget uuids altogether?
Thanks.
EDIT/UPDATE per #rolf:
Since I just discovered that the django shell differs more than I realized (the shell can find settings, the regular interpreter can't) I decided to run this once in each one, but the results were the same.
(cannon)malikarumi#Tetuoan2:~/Projects/cannon/jamf$ python manage.py shell
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
IPython 4.0.3 -- An enhanced Interactive Python.
In [1]: uuid.UUID(a82857b6-e336-4c6c-8499-47601770b39d)
File "<ipython-input-1-e282858da374>", line 1
uuid.UUID(a82857b6-e336-4c6c-8499-47601770b39d)
^
SyntaxError: invalid syntax
In [2]: uuid.UUID(a0a69415-6627-43db-8c7a-b57d0c4cefe2)
File "<ipython-input-2-befebf1573ba>", line 1
uuid.UUID(a0a69415-6627-43db-8c7a-b57d0c4cefe2)
^
SyntaxError: invalid syntax
In [3]: uuid.UUID(e6e11b06-ea3b-4e98-a31f-9a83447ad884)
File "<ipython-input-3-a59ea095e61a>", line 1
uuid.UUID(e6e11b06-ea3b-4e98-a31f-9a83447ad884)
^
SyntaxError: invalid syntax
In [4]: uuid.UUID(bd116432-65d7-4612-abfe-9a99dcaf5cad)
File "<ipython-input-4-c4a04434aa3c>", line 1
uuid.UUID(bd116432-65d7-4612-abfe-9a99dcaf5cad)
^
SyntaxError: invalid syntax
Now that I have posted this, I notice that even Stack Overflow treats these uuid differently, i.e., the way they are colored, if that's relevant and meaningful here.
But now that we know this, what do we do with / about it?
2nd Update
This morning I thought, what about a uuid that had never been anywhere but in Django? So here's what I did:
In [5]: e.uuid
Out[5]: UUID('61877565-5fe5-4175-9f2b-d24704df0b74')
In [6]: uuid.UUID(61877565-5fe5-4175-9f2b-d24704df0b74)
File "<ipython-input-6-56137f5f4eb6>", line 1
uuid.UUID(61877565-5fe5-4175-9f2b-d24704df0b74)
^
SyntaxError: invalid syntax
In [7]: uuid.UUID('61877565-5fe5-4175-9f2b-d24704df0b74')
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-7-3b4d3e5bd156> in <module>()
----> 1 uuid.UUID('61877565-5fe5-4175-9f2b-d24704df0b74')
NameError: name 'uuid' is not defined
This is apparently because I left the quote around the alphanumeric, but why that would generate a uuid not defined error, instead of 'string type' or some such error is beyond me.
In [8]: uuid.UUID(61877565-5fe5-4175-9f2b-d24704df0b74)
File "<ipython-input-8-56137f5f4eb6>", line 1
uuid.UUID(61877565-5fe5-4175-9f2b-d24704df0b74)
^
SyntaxError: invalid syntax
The first time I keyed in the characters by hand. I decided to repeat the test by copying and pasting, but as you can see, it made no difference. If there was something weird about the way only the 5 that the caret is pointing to was generated, we might be on to something, but if so, why do I get the same error in the same place when I typed it in by hand myself?
This no longer seems like a json issue to me, since – as far as I know – json has never touched this uuid, unless it did somehow in the internal workings of Django.
Instead, there is either
1. something wrong with the way uuid.UUID generates uuids, or
2. the way it generates them on my system, (Ubuntu 15.10, Django 1.9.1, Python 2.7.10) or
3. the way it reads and evaluates them when they come back, like in uuid.UUID() or being input outside the internal, automatic uuid generation process.
But that also means people using uuid.UUID() to generate uuids will never know there is an issue unless they do what I did, which is try to bring them in from outside. I remember reading somewhere that all uuids are supposed to be compatible. So, unless someone here has a better insight, I think we might be up for a bug report. But is it a Python bug, a Django bug, or both?
Your syntax is wrong:
uuid.UUID('61877565-5fe5-4175-9f2b-d24704df0b74') # note the quotes
When I use AppleScript to get the properties of an object, a record is returned.
tell application "iPhoto"
properties of album 1
end tell
==> {id:6.442450942E+9, url:"", name:"Events", class:album, type:smart album, parent:missing value, children:{}}
How can I iterate over the key/value pairs of the returned record so that I don't have to know exactly what keys are in the record?
To clarify the question, I need to enumerate the keys and values because I'd like to write a generic AppleScript routine to convert records and lists into JSON which can then be output by the script.
I know it's an old Q but there are possibilities to access the keys and the values now (10.9+). In 10.9 you need to use Scripting libraries to make this run, in 10.10 you can use the code right inside the Script Editor:
use framework "Foundation"
set testRecord to {a:"aaa", b:"bbb", c:"ccc"}
set objCDictionary to current application's NSDictionary's dictionaryWithDictionary:testRecord
set allKeys to objCDictionary's allKeys()
repeat with theKey in allKeys
log theKey as text
log (objCDictionary's valueForKey:theKey) as text
end repeat
This is no hack or workaround. It just uses the "new" ability to access Objective-C-Objects from AppleScript.
Found this Q during searching for other topics and couldn't resist to answer ;-)
Update to deliver JSON functionality:
Of course we can dive deeper into the Foundation classes and use the NSJSONSerialization object:
use framework "Foundation"
set testRecord to {a:"aaa", b:"bbb", c:"ccc"}
set objCDictionary to current application's NSDictionary's dictionaryWithDictionary:testRecord
set {jsonDictionary, anError} to current application's NSJSONSerialization's dataWithJSONObject:objCDictionary options:(current application's NSJSONWritingPrettyPrinted) |error|:(reference)
if jsonDictionary is missing value then
log "An error occured: " & anError as text
else
log (current application's NSString's alloc()'s initWithData:jsonDictionary encoding:(current application's NSUTF8StringEncoding)) as text
end if
Have fun, Michael / Hamburg
If you just want to iterate through the values of the record, you could do something like this:
tell application "iPhoto"
repeat with value in (properties of album 1) as list
log value
end repeat
end tell
But it's not very clear to me what you really want to achieve.
Basically, what AtomicToothbrush and foo said. AppleScript records are more like C structs, with a known list of labels, than like an associative array, with arbitrary keys, and there is no (decent) in-language way to introspect the labels on a record. (And even if there were, you’d still have the problem of applying them to get values.)
In most cases, the answer is “use an associative array library instead.” However, you’re specifically interested in the labels from a properties value, which means we need a hack. The usual one is to force an error using the record, and then parse the error message, something like this:
set x to {a:1, b:2}
try
myRecord as string
on error message e
-- e will be the string “Can’t make {a:1, b:2} into type string”
end
Parsing this, and especially parsing this while allowing for non-English locales, is left as an exercise for the reader.
ShooTerKo's answer is incredibly helpful to me.
I'll bring up another possibility I'm surprised I didn't see anyone else mention, though. I have to go between AppleScript and JSON a lot in my scripts, and if you can install software on the computers that need to run the script, then I highly recommend JSONHelper to basically make the whole problem go away:
https://github.com/isair/JSONHelper