I have created a Symfony Bundle -- using the DependencyInjection-mechanism that Symfony4 provides.
When my custom Extension is initialized, the provided parameters from my_extension.yaml get piped into my "load"-method as expected, as described in the SymfonyDocs ( https://symfony.com/doc/current/bundles/configuration.html ).
However, if I add an additional yaml-file into the designated package-scope (e.g. config/packages/dev/my_extension.yaml), these parameters end up being merged by the built-in processConfiguration() - method in a strange way:
Whereas the first config is parsed correctly and all parameter keys are kept intact, the second file is not merged as expected, but all the contained values get transformed into numeric array keys, i.e. the original parameter keys get lost on the way.
Example:
Contents of config/packages/my_extension.yaml
my_extension:
parameters:
some_attribute: "original_value1"
some_other_attribute: "original_value2"
Contents of config/packages/dev/my_extension.yaml
my_extension:
parameters:
some_attribute: "new_value1"
specific_attribute: "new_value3"
Results in a merged configuration array that looks like this
parameters:
some_attribute: "new_value1"
some_other_attribute: "new_value2"
0: "new_value1"
1: "new_value2"
2: "new_value3"
while I would expect the resulting configuration to be
parameters:
some_attribute: "new_value1"
some_other_attribute: "original_value2"
specific_attribute: "new_value3"
The last (correct) result is what I get if I manually merge the configs in the "load"-method of my extension like this:
$mergedConfig = [];
foreach($configs as $config) {
$mergedConfig = array_replace_recursive($mergedConfig, $config);
}
$config = $this->processConfiguration($configuration, [$mergedConfig]);
However, why can't I rely on the built-in merging strategy that Symfony4 provides for this scenario? Is this a bug or did I get anything wrong about how Symfony is supposed to merge parameters from different Config-sources?
I am new to Robot Framework and am trying to validate the contents of some JSON that is returned from a web service. The problem is that some attributes of the json objects have dashes in them and Robot doesn't seem to like this. I have something like the following
&{deployment} = list deployment ${deployment_name}
&{changeSets} = Set Variable ${deployment.ChangeSets}
&{myChangeSet} = Set Variable ${changeSets.my-change-set}
Should Be True ${myChangeSet.UseLocal}
Should Be Equal As Strings ${myChangeSet.Version} ${update_version}
But Robot fails on the 3rd line with the following error:
Resolving variable '${changeSets.my-change-set}' failed: AttributeError: my
I tried to escape the dashes but that still doesn't seem to work:
Resolving variable '${changeSets.my\-change\-set}' failed: SyntaxError: unexpected character after line continuation character (<string>, line 1)
I can't seem to find any information in the Robot docs with other ways to retrieve dict keys outside of the dot-notation. Any suggestions?
The use of dot notation is just a convenience. You can still access them the normal way (documented in the dictionary variables section of the user guide as &{NAME}[key]):
&{changeSets}[my-change-set]
Or, with extended variable syntax, which treats everything inside {} as a python expression:
${changeSets['my-change-set']}
Here is a working example illustrating these two methods:
*** Variables ***
&{changeSets} my-change-set=foo
*** Test Cases ***
Test 1
should be equal ${changeSets['my-change-set']} foo
Test 2
should be equal &{changeSets}[my-change-set] foo
Im trying to write a small app that retrieves a JSON file (it contains a list of items, which all have some properties), saves its contents to the DB and then displays some of it later on. I have Zotonic up and running, and generating some HTML is no problem.
ATM i'm stuck trying to figure out how to define a custom resource and how to get the data from the JSON in the DB. When the data is there I should be fine, that part seems covered ok by the documentation.
I wrote some standalone erlang scripts that fetch the data and I noticed that Zotonic has a library for decoding JSON so that part should be fine. Any tips on where to put which code or where to look further?
The z_db module allows for creating custom tables by using:
z_db:create_table(Table, Cols, Context).
The Table variable is your table name which can be either an atom or a list containing a single atom.
The Cols is a list of column definitions, which are defined by records. Currently the record definition (you can find this in include/zotonic.hrl) is:
-record(column_def, {name, type, length, is_nullable=true, default, primary_key}).
See Erlang docs on records for more info on records
Example code which I put in users/sites/[sitename]/models/m_[sitename].erl:
init(Context) ->
case z_db:table_exists(?table,Context) of
false ->
z_db:create_table(tablename,
[
#column_def{name=id, type="serial"},
#column_def{name=gid, type="integer", is_nullable=false},
#column_def{name=magnitude, type="real"},
#column_def{name=depth, type="real"},
#column_def{name=location, type="character varying"},
#column_def{name=time, type="integer"},
#column_def{name=date, type="integer"}
], Context);
true -> ok
end,
ok.
Pay attention to what options of the record you specify. Most of the errors I got were e.g. from specifying a length on the integer fields.
The models/m_sitename:init/1 does not get called on site start. The sitename:init/1 does get called so I call the init function there to ensure the table exists. Example:
init(Context) ->
m_sitename:init(Context).
It is called by Zotonic with the Context variable of the site automatically. You can get this variable manually as well with z:c(sitename).. So if you call the m_sitename:init(Context). from somewhere else you would do:
m_sitename:init(z:c(sitename)).
Next, insertion in the DB can be done with:
z_db:insert(Table, PropList, Context).
Where Table is again an atom or a list containing a single atom representing the table name. Context is the same as above.
PropList is a property list which is a list containing tuples consisting of two elements where the first is an atom and the second is its associated value/property. Example:
PropList = [
{row, Value},
{anotherrow, AnotherValue}
].
Table = tablename.
Context = z:c(sitename).
z_db:insert(Table, PropList, Context).
See Erlang docs on Property Lists for more info on property lists.
=== The dependencies have been updated so if you build from source the step directly below is no longer needed ===
The JSON part is bit more tricky. Included with Zotonic are mochijson2 and as a secondary dependency also jiffy. The latest version of jiffy contains jiffy:decode/2 which allows you to specify maps as a return type. Much more readable than the standard {struct, {struct, <<"">>}} monster. To update to the latest version edit the line in deps/twerl/rebar.config that says
{jiffy, ".*", {git, "https://github.com/davisp/jiffy.git", {tag, "0.8.3"}}},
to
{jiffy, ".*", {git, "https://github.com/davisp/jiffy.git", {tag, "0.14.3"}}},
Now run z:m(). in the Zotonic shell. (you must do this after every change in your code).
Now check in the Zotonic shell if there is a jiffy:decode/2 available by typing jiffy: <tab>, it will show a list of available functions and their arity.
To retrieve a JSON file from the internet run:
{ok, {{_, 200, _}, _, Body}} = httpc:request(get, {"url-to-JSON-here", []}, [], [])
Which will yield the variable Body with the contents. See Erlang docs on http client for more info on this call.
Next convert the contents of Body to Erlang terms with:
JsonData = jiffy:decode(Body, [return_maps]).
What you have to do next depends a lot on the structure of your JSON resource. Keep in mind that everything is now in binary UTF-8 encoded strings! If you print JsonData to screen (just enter JsonData. in your Zotonic/Erlang shell) you will see a lot of #map{<<"key"", <<"Value">>} this.
My data was nested so I had to extract the needed data like this:
[{_,ItemList}|_] = ListData.
This gave me a list of maps, and in order to deal with them as individual items I used the following function:
get_maps([]) ->
done;
get_maps([First|Rest]) ->
Map = maps:get(<<"properties">>, First),
case is_map(Map) of
true ->
map_to_proplist(Map),
get_maps(Rest);
false -> done
end,
done;
get_maps(_) ->
done.
As you might remember, the z_db:insert/3 function needs a property list to populate rows, so that what the call to map_to_proplist/1 is for. How this function looks is completely dependent on how your data looks but as an example here is what worked for me:
map_to_proplist(Map) ->
case is_map(Map) of
true ->
{Value1,_} = string:to_integer(binary_to_list(maps:get(<<"key1">>, Map))),
{Value2,_} = string:to_float(binary_to_list(maps:get(<<"key2">>, Map))),
{Value3,_} = string:to_float(binary_to_list(maps:get(<<"key3">>, Map))),
Value4 = binary_to_list(maps:get(<<"key4">>, Map)),
{Value5,_} = string:to_integer(binary_to_list(maps:get(<<"key5">>, Map))),
{Value6,_} = string:to_integer(binary_to_list(maps:get(<<"key6">>, Map))),
PropList = [{rowname1, Value1}, {rowname2, Value2}, {rowname3, Value3}, {rowname4, Value4}, {rowname5, Value5}, {rowname6, Value6}],
m_sitename:insert_items(PropList,z:c(sitename)),
ok;
false ->
ok
end.
See the documentation on string:to_list/1 as to why the tuples are needed when casting. The call to m_sitename:insert_items(PropList,z:c(sitename)) calls the z_db:insert/3 in models/m_sitename.erl but wrapped in a catch:
insert_items(PropList,Context) ->
(catch z_db:insert(?table, PropList, Context)).
Ok, quite a long post but this should get you up and running if you were looking for this answer.
The above was done with Zotonic 0.13.2 on Erlang/OTP 18.
A repost (except the JSON part) of my post in the Zotonic Developers group.
I've recently added the Sphider crawler to my site in order to add search functionality. But the default search.php that comes with the distribution of Sphider that I downloaded is too plain and doesn't integrate well with the rest of my site. I have a little navigation bar at the top of the site which has a search box in it, and I'd like to be able to access Sphider's search results through that search field using Ajax. To do this, I figure I need to get Sphider to return its results in JSON format.
The way I did that is I used a "theme" that outputs JSON (Sphider supposts "theming" its output). I found that theme on this thread on Sphider's site. It seems to work, but more strict JSON parsers will not parse it. Here's some example JSON output:
{"result_report":"Displaying results 1 - 1 of 1 match (0 seconds) ", "results":[ { "idented":"false", "num":"1", "weight":"[100.00%]", "link":"http://www.avtainsys.com/articles/Triple_Contraints", "title":"Triple Contraints", "description":" on 01/06/12 Project triple constraints are time, cost, and quality. These are the three constraints that control the performance of the project. Think about this triple-constraint as a three-leg tripod. If one of the legs is elongated or", "link2":"http://www.avtainsys.com/articles/Triple_Contraints", "size":"3.3kb" }, { "num":"-1" } ], "other_pages":[ { "title":"1", "link":"search.php?query=constraints&start=1&search=1&results=10&type=and&domain=", "active":"true" }, ] }
The issue is that there is a trailing comma near the end. According to this, "trailing commas are not allowed" when using PHP's json_decode() function. This JSON also failed to parse using this online formatter. But when I took the comma out, it worked and I got this better-formatted JSON:
{
"result_report":"Displaying results 1 - 1 of 1 match (0 seconds) ",
"results":[
{
"idented":"false",
"num":"1",
"weight":"[100.00%]",
"link":"http://www.avtainsys.com/articles/Triple_Contraints",
"title":"Triple Contraints",
"description":" on 01/06/12 Project triple constraints are time, cost, and quality. These are the three constraints that control the performance of the project. Think about this triple-constraint as a three-leg tripod. If one of the legs is elongated or",
"link2":"http://www.avtainsys.com/articles/Triple_Contraints",
"size":"3.3kb"
},
{
"num":"-1"
}
],
"other_pages":[
{
"title":"1",
"link":"search.php?query=constraints&start=1&search=1&results=10&type=and&domain=",
"active":"true"
}
]
}
Now, how would I do this programmatically? And (perhaps more importantly), is there a more elegant way of accomplishing this? And you should know that PHP is the only language I can run on my shared hosting account, so a Java solution for example would not work for me.
In search_result.html, you can surround the , at the end of the foreach loop with condition to only print if the index is strictly less than the number of pages - 1.
I want to formally define a schema for a JSON-based protocol.
I have two criteria for the schema:
1. I want to be able to use tools to create parser/serializer (php and .net).
2. Result JSON should be easy to human-read
Here is the context. The schema will be describing a game character, as an example I will take one aspect of the profile - professions.
A character can have up to 2 professions (from a list of 10), each profession is described by a name and a level, e.g.:
Skinning - level 200
Blacksmith - level 300
To satisfy criterion #1 it really helps to have XSD schema (or JSON Schema) to drive code generator or a parser library. But that means that my JSON must look something like:
character : {
professions : [
{ profession : "Skinning", level : 525 }
{ profession : "Blacksmith", level : 745 }
]
}
but it feels too chatty, I would rather have JSON look like (notice that profession is used as a key):
character {
professions : {
"Skinning" : 525,
"Blacksmith" : 745
}
}
but the later JSON can not be described with XSD without having to define an element for each profession.
So I am looking for a solution for my situation, here are options I have identified:
shut up and make JSON XSD-friendly (first snippet above)
shut up and make JSON human-friendly and hand-code the parser/serializer.
but I would really like to find a solution that would satisfy both criteria.
Note: I am aware that Newton-King's JSON library would allow me to parse professions as a Dictionary - but it would require me to hand code the type to map this JSON to. Therefore so far I am leaning towards option #2, but I am open to suggestions.
Rename profession to name so it'd be like:
character : {
professions : [
{ name : "Skinning", level : 525 }
{ name : "Blacksmith", level : 745 }
]
}
Then after it's serialized on the client model would be like this:
profession = character.professions[0]
profession.name
=> "Skinning"
Your options are, as you said...
1 shut up and use xml
2 shut up and build your own
OR maybe 3... http://davidwalsh.name/json-validation
I would do #1
- because xml seems to be a fairly common way to transform stuff from X => Y formats
- I prefer to work in C# not JS
- many people use XML, its an accepted standard, there are many resources out there to help you along the way