In Airflow GUI can see that my xcom variable looks like ['val1', 'val2', 'val3']. If I have understood it correctly from the documentation, when I do {{ ti.xcom_pull("task_id") }} will be a string. So my question is how do I split the xcom varible so that I get three seperate values wihch I will then use later in my other task?
Your issue is that you have a string representation of list "['val1', 'val2', 'val3']" and you want to convert it into a list object: ['val1', 'val2', 'val3'] so you can do it in any string to list manipulation that you prefer.
However, Airflow has a better option. You can set render_template_as_native_obj=True :
dag = DAG(
...
render_template_as_native_obj=True,
)
by setting render_template_as_native_obj=True Airflow uses Jinja’s NativeEnvironment, which allows rendering a template produces a native Python type. You can read about it in this doc.
Related
I am currently working with the DatabricksSubmitRunOperator in Airflow. When I tried to integrate the jinja template into the operator, by using it in the "json"-parameter of the operator, I was facing an error. The problem is, that jinja returns a string but Databricks operator needs a dict type.
I already looked up the source code of the operator. "json" seems to be a template_field, which should be fine.
Is there a way to make jinja return a dict type instead of a string in this case? Or maybe another workaround?
Thanks in advance!
There is the new feature in Airflow 2.1/0 that can help with that https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects - it uses JINJA capability of "safe evaluation" of the string and returning and object that the string represents directly in your code.
Is there a way to read a column of doctrine type "simply_array" or "array" in json?
My doctrine database is approached from another api and I want to read data from that api. However there is a column of type doctrine array that I want to convert into JSON.
I am unsure if there is a preferred way of doing this or I need to hack my way around it.
Here is an example of what is stored in the database as a doctrine array:
"a:1:{i:0;a:3:{s:3:\u0022day\u0022;i:5;s:4:\u0022time\u0022;s:7:\u0022morning\u0022;s:12:\u0022availability\u0022;N;}}"
That looks like the format of PHP's serialize() function. And the literal double-quotes in the string have been converted to unicode escape sequences.
You could do the following:
Fetch the serialized string
Fix the \u0022 sequences (replace them with ")
unserialize() it to reproduce the array
Convert the array to JSON with json_encode().
I am using json.load() to read a json config file into a dictionary, but I couldn't use is keyword for comparing string values.
config = json.load(open(config_file))
if config['some_key'] is 'abc':
# code block
even the value of config['some_key'] is abc, the if block is passed somehow. But if I changed the code to,
if config['some_key'] == 'abc':
# code block
the if code body will get executed. I am wondering what is the issue. Since string is a python object, so should be no problem using is for value comparison.
I'm using webwrite to post to an api. One of the field names in the json object I'm trying to setup for posting is odata.metadata. I'm making a struct that looks like this for the json object:
json = struct('odata.metadata', metadata, 'odata.type', type, 'Name', name,);
But I get an error
Error using struct
Invalid field name "odata.metadata"
Here's the json object I'm trying to use in Matlab. All strings for simplicity:
{
"odata.metadata": "https://website.com#Element",
"odata.type": "Blah.Blah.This.That",
"Name": "My Object"
}
Is there a way to submit this json object or is it a lost cause?
Field names are not allowed to have dots in them. The reason why is because this will be confused with accessing another nested structure within the structure itself.
For example, doing json.odata.metadata would be interpreted as json being a struct with a member whose field name is odata where odata has another member whose field name is metadata. This would not be interpreted as a member with the combined field name as odata.metadata. You're going to have to rename the field to something else or change the convention of your field name slightly.
Usually, the convention is to replace dots with underscores. An automated way to take care of this if you're not willing to manually rename the field names yourself is to use a function called matlab.lang.makeValidName that takes in a string and converts it into a valid field name. This function was introduced in R2014a. For older versions, it's called genvarname.
For example:
>> matlab.lang.makeValidName('odata.metadata')
ans =
odata_metadata
As such, either replace all dots with _ to ensure no ambiguities or use matlab.lang.makeValidName or genvarname to take care of this for you.
I would suggest using a a containers.Map instead of a struct to store your data, and then creating your JSON string by iterating over the Map filednames and appending them along with the data to your JSON.
Here's a quick demonstration of what I mean:
%// Prepare the Map and the Data:
metadata = 'https://website.com#Element';
type = 'Blah.Blah.This.That';
name = 'My Object';
example_map = containers.Map({'odata.metadata','odata.type','Name'},...
{metadata,type,name});
%// Convert to JSON:
JSONstr = '{'; %// Initialization
map_keys = keys(example_map);
map_vals = values(example_map);
for ind1 = 1:example_map.Count
JSONstr = [JSONstr '"' map_keys{ind1} '":"' map_vals{ind1} '",'];
end
JSONstr =[JSONstr(1:end-1) '}']; %// Finalization (get rid of the last ',' and close)
Which results in a valid JSON string.
Obviously if your values aren't strings you'll need to convert them using num2str etc.
Another alternative you might want to consider is the JSONlab FEX submission. I saw that its savejson.m is able to accept cell arrays - which can hold any string you like.
Other alternatives may include any of the numerous Java or python JSON libraries which you can call from MATLAB.
I probably shouldn't add this as an answer - but you can have '.' in a struct fieldname...
Before I go further - I do not advocate this and it will almost certainly cause bugs and a lot of trouble down the road... #rayryeng method is a better approach
If your struct is created by a mex function which creates a field that contains a "." -> then you will get what your after.
To create your own test see the Mathworks example and modify accordingly.
(I wont put the full code here to discourage the practice).
If you update the char example and compile to test_mex you get:
>> obj = test_mex
obj =
Doublestuff: [1x100 double]
odata.metadata: 'This is my char'
Note: You can only access your custom field in Matlab using dynamic fieldnames:
obj.('odata.metadata')
You need to use a mex capability to update it...
Is this possible to have a node property as json raw string and to filter on it with cypher ?
I have a node with some defined properties and metadata (json raw string).
I would like to select or filter on those metadata property.
This is something like this :
START movie=node:TYPE_INDEX(Type = 'MOVIE') // Start with the reference
MATCH movie-[t:TAG]->tag
WHERE collect(movie.Metadata).RatingPress > 3
RETURN distinct movie.Label
And metadata are something like this :
{"RatingPress" : "0","RatingSpectator" : 3"}
I have expected to use collect function in order to call the property like this :
collect(movie.Metadata).RatingPress
But, of course it fails...
Is this a way to bind some json string from a node property with cypher ?
Thanks for your help
That's going against the principles of properties. Why not set the properties in the JSON metadata directly on the node?
But to answer your question:
No, cypher has no knowledge about JSON.
We treat the entire Node as a JSON blob. Since Neo4j doesn't support hierarchical properties, we flatten out the JSON into delimited property names on save and unflatten them on read. You can then form Cypher queries on (for example) property name "foo.bar.baz". The queries tend to look a bit funky because you'll need to quote them using single back quotes, but it works.