I'm using Scrapy to extract some values from a website. One of the fields is "price" and I have something like:
...
item = {
"title": title,
"url" : url,
"price": {
"symbol": the_currency_symbol,
"amount": float_number
}
}
yield item
I set the output to be a JSON file and I yield a dictionary item.
The problem is that when I open the output JSON with the items I see this:
{
"title": title,
"url" : url,
"price": {
"symbol": "\u00a3",
"amount": 12.99
}
}
How can I see the correct currency symbol in the JSON file?
Scrapy normally produces JSON feeds in ASCII encoding. Your JSON file has correct data but to see the currency symbol properly you can convert the output JSON file to UTF-8 encoding.
You can make Scrapy generate JSON in UTF-8 encoding by setting FEED_EXPORT_ENCODING="utf-8". For more help, see the answers to the question: Scrapy json response convert in utf-8 encode and Scrapy documentation https://docs.scrapy.org/en/latest/topics/feed-exports.html#std-setting-FEED_EXPORT_ENCODING.
If you do not want to run the scraper again then you can use some tool like https://stedolan.github.io/jq/ on your JSON file like jq file.json > outfile.json. Then outfile.json will have proper symbols.
Related
Trying to load JSON file from s3 into Redshift using Copy with JSONPATHS file. The file contains N number of records.
Loading the entire set in one go throws an error:
Invalid operation: Invalid JSONPath format. Supported notations are 'dot-notation' and 'bracket-notation'
The Json paths:
{"jsonpaths":
[
"$.item[:].col1",
"$.item[:].col2",
"$.item[:].col3"
]
}
sample file:
{"item":
[
{
"col1":"A",
"col2":"b",
"col3":"d"
},
{
"col1": "123",
"col2": "red",
"col3": "456"
}
]
}
Working FILE:-
{"jsonpaths":
[
"$.item[0].col1",
"$.item[0].col2",
"$.item[0].col3"
]
}
What am I doing wrong to cause this error?
As per the documentation, there are 2 ways of specifying the JSONPaths. One is to use the dot notation and another is to use the bracket notation.
In this example, the user has used the dot notation, but the arrays have been indexed using a colon (:). The correct way to index JSON arrays elements is to use a number. Hence the second example of the JSONPath file works.
I want to read texts from a json file and then convert it to a python dict object, but meet some problems when the backslash is in the json data.
what my json file is like:
{
"foo": "+|\\*|/",
}
I want to get a dict like this:
{
"foo": "+|\*|/",
}
but I get this actually:
{
"foo": "+|\\*|/",
}
If I change the json file like this:
{
"foo": "+|\*|/"
}
or
{
"foo": "+|\\\*|/",
}
then I will get an error.
The python versoin is 3.8.0
The problem may be not in the loading of the JSON file, but in displaying its content as Python object (since repr() is used under the hood, and returns the representation with single quotes and escaped characters).
For example, my file (named 1.json) is
{
"foo": "+|\\*|/"
}
I'm using the following code to load it in Python 3.8.0:
import json
a = json.loads(open("1.json").read())
Now print(a) prints {'foo': '+|\\*|/'}, while print(a["foo"]) will print the correct result: +|\*|/.
So, please check that your displayed part is not using repr().
I have the following JSON file.
{
"reviewerID": "ABC1234",
"productID": "ABCDEF",
"reviewText": "GOOD!",
"rating": 5.0,
},
{
"reviewerID": "ABC5678",
"productID": "GFMKDS",
"reviewText": "Not bad!",
"rating": 3.0,
}
I want to parse without SparkSQL and use a JSON parser.
The result of parse that i want is textfile.
ABC1234::ABCDEF::5.0
ABC5678::GFMKDS::3.0
How to parse the json file by using json parser in spark scala?
tl;dr Spark SQL supports JSONs in the format of one JSON per file or per line. If you'd like to parse multi-line JSONs that can appear together in a single file, you'd have to write your own Spark support as it's not currently possible.
A possible solution is to ask the "writer" (the process that writes the files to be nicer and save one JSON per file) that would make your life much sweeter.
If that does not give you much, you'd have to use mapPartitions transformation with your parser and somehow do the parsing yourself.
val input: RDD[String] = // ... load your JSONs here
val jsons = jsonRDD.mapPartitions(json => // ... use your JSON parser here)
I'm passing some data through Json array to an html table. My problem is that the encoding of some frensh letter like (é, è, à),
my json array is like :
paniersFromJsonFile = [
{
"reference": "62010",
"LibelleDeLaPiece": "BOUCLIER",
"origine": "Pièce d'origine",
"distributeur": "datasier"
},
{
"reference": "60100",
"LibelleDeLaPiece": "Grille",
"origine": "Pièce d'origine",
"distributeur": "mvc"
}]
in the screen i have this
Thanks
yes,you need to save file as UTF-8(As gautam said) without BOM (Byte Order Mark in the begining of the file), Notepad++ allows that option
I have a 3rd party service which returns me the following response:
JSON 1
{"Bag":{"Type":{"$":"LIST"},"Source":{"$":"ABC"},"Id":{"$":"151559458"},"Name":{"$":"Bag list"},"Source":{"$":"ABC"},"CustomerId":{"$":"abc#gmail.com"},"DateTime":{"$":"2014-07-17T12:36:01Z"}}}
But I have to format this JSON into the following format:
JSON2
{"Bag":{"Type":"LIST","Source":"ABC","Id":"151559458","Name":"Bag list","Source":"ABC","CustomerId":"abc#gmail.com","DateTime":"2014-07-17T12:36:01Z"}}
And Vice versa like from client I get JSON2 and I have to send this response to service in JSON1 format.
Given the input in the first JSON schema, transform it to a language specific data structure. You can typically find a library to do this.
Transform your language specific data structure, using the idioms of whichever language you are using, to the equivalent of the second JSON schema.
Transform the language specific data structure to JSON text. You can typically find a library to do this.
You can use jq tool http://stedolan.github.io/jq
Then the conversion is console one-liner:
$ jq '{Bag: .Bag | with_entries({key, value: .value."$"})}' file.json
And the result is
{
"Bag": {
"Type": "LIST",
"Source": "ABC",
"Name": "Bag\nlist",
"Id": "151559458",
"DateTime": "2014-07-17T12:36:01Z",
"CustomerId": "abc#gmail.com"
}
}