I am requesting a web service and getting a JSON data as shown below. However, I keep getting the following error:
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{feff}\x{feff}{"ur...") at /usr/share/perl5/JSON/Any.pm
http request:
Date: Tue, 16 Apr 2013 10:41:03 GMT
Server: nginx/0.7.67
Content-Type: application/json; charset=utf-8
Client-Date: Tue, 16 Apr 2013 10:41:03 GMT
Client-Peer: 127.0.1.1:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
json data:
{"url":"http:\/\/example.com\/service\/rest.htm?req_data=<auth_req><request_token>20130416f186a9c0480e2501e73d19dbcd79d354<\/request_token> <\/auth_req>&user=208860&service=auth.execute&sid=0001&format=xml&v=2.0& sign=pn9xjQjzTgQuAMarLDtiZCMaGZm4bSo8aUTGtkSt1GrxPGtK29oIL1DgHveVMwf2n7rxLHzyWrNd%2BYU6%2BxZCzs56JkMtxQMPxEJ%2Bu9Eqk5SRL6EAjWMeKheix5frPyHi0hQ4nnbiVm%2Bx3bF0KFq3cORvVCeq8wBoZU1HQXD%2BuuY%3D"}
I suspect some kind of encoding issue because the JSON string validates fine in jslint JSON validator. But I don't know what else to look for. Please help, thanks.
\x{feff} is a BOM (Byte Order Mark). I am not sure whether it is allowed at the beginning of a JSON, but it definitely should not be repeated.
thanks to choroba for prompting me to look for the BOM. Greped 3rd party library files which generates the url and surely found the BOM in them.
grep -rl $'\xEF\xBB\xBF' . # Got BOM?
perl -pi -nle 's/^\xEF\xBB\xBF//' *.lib # remove them!
cheers.
Related
We have developed a webservice having CICS as the HTTP SERVER (service provider). This Webservice takes the input JSON (which has both English and Chinese characters) from any client/POSTMAN tool and will be processed in Mainframe (CICS).
DFHJS2LS: JSON schema to high-level language conversion for request-response services
We are using this proc - DFHJS2LS to enable webservices in Mainframe. ThisI BM provided procedure does the conversion of JSON to MF copybook and vice-versa. Also it converts the UTF-8 code unit into UTF-16 when it reaches mainframe copybook.
Issue:
The issue what we face now is on the Chinese characters. The Chinese characters which we pass in JSON are not getting converted properly and they are getting corrupted when it is received inside mainframe. The conversion from UTF-8 to UTF-16 is not happening (this is my suspect).
市 - this is the chinese character passed in JSON (POSTMAN).
Expected value in Mainframe copybook is 5E02(UTF-16 - hex value)
but we got 00E5 00B8 0082(UTF-8 hex value)
we have tried all header values and still no luck.....
content type = application/json
charset=UTF-8 / UTF-16
Your inputs are much appreciated in addressing this DBCS/unicode/chinese character issue.
In the COBOL are you declaring the filed that will receive the Chinese characters as Pic G :
01 China-Test-Message.
03 Msg-using-pic-x Pic X(10).
03 Msg-using-pic-g Pic G(4) Usage Display-1.
Try "USAGE NATIONAL" which shoul dmap to UTF-16 which is probably the code page for the chinese character.
RTFM here:-
https://www.ibm.com/support/knowledgecenter/SS6SG3_6.3.0/pg/concepts/cpuni01.html
The chinese conversion is resolved once we changed our HTTP header to this -
Content-Type = application/json;charset=UTF-8
thanks everyone for the support.
so I'm trying to use and Arduino Due and a SIM7600 LTE Shield to send a GET request to a server. I've tried multiple servers to no avail and I'm not really sure what I'm doing wrong. Below are my AT commands
19:34:00.607 -> AT+CHTTPACT="website.co.uk",80
19:34:00.710 -> +CHTTPACT: REQUEST
19:34:07.533 -> GET website.co.uk/4gtest.php HTTP/1.0
19:34:12.302 -> Host: website.co.uk
19:34:19.101 -> Content-Length: 42
19:34:28.000 ->
19:34:28.000 -> OK
And below is the response:
19:34:28.581 -> +CHTTPACT: DATA,295
19:34:28.581 -> http/1.1 400 bad request
19:34:28.581 -> server: nginx
19:34:28.581 -> date: tue, 04 feb 2020 19:34:27 gmt
19:34:28.581 -> content-type: text/html
19:34:28.581 -> content-length: 150
19:34:28.581 -> connection: close
19:34:28.581 ->
19:34:28.581 -> <html>
19:34:28.581 -> <head><title>400 Bad Request</title></head>
19:34:28.615 -> <body>
19:34:28.615 -> <center><h1>400 Bad Request</h1></center>
19:34:28.615 -> <hr><center>nginx</center>
19:34:28.615 -> </body>
19:34:28.615 -> </html>
19:34:28.648 ->
19:34:28.648 -> +CHTTPACT: 0
There is definitely an internet connection as it returns custom error pages from the servers but I'm not sure why it can't get the pages I want.
Any help would really be appreciated
Thanks
I've been able to get a server response with the following request:
GET /4gtest.php HTTP/1.1<CR><LF>
Host: website.co.ukCR><LF>
<CR><LF>
<CR><LF>
I got Error 404 response, probably because it was a test page that had currently been removed. Anyway it is not Error 400 response meaning that, at least, the request is not malformed.
Some description about the request:
After GET command, only the path is expected (as correctly suggested by user #juraj's comment)
Since your error response specified HTTP/1.1, I used the same version for my request
The hostname is passed in the line Host: website.co.uk
Please note that each line is terminated with the <CR> + <LF> couple of characters (carriage return, 0x13 ASCII, and line feed, 0x10 ASCII). It is important to specify it because in your question it is not clear how the lines are terminated
Also note that after the last line a double <CR> + <LF> couple terminates the header section of the request
About Content-Length
I have omitted Content-Length field in my request. As brilliantly explained in the answers to this question, this field specifies the number of octects contained in the message body, after the header. But:
The AT command log in your question did not mention any body after the header
As explained in this answer, request bodies in GET requests are not explicitely forbidden, but they are not recommended
Yes, you can send a request body with GET but it should not have any
meaning. If you give it meaning by parsing it on the server and
changing your response based on its contents, then you are ignoring
this recommendation in the HTTP/1.1 spec, section 4.3
I need to retrieve some parts of a message which looks like
Subject: Test message
Message-Id: <2788db2f-b5c3-4b8c-881e-362c3df4f915#server.local>
Mime-Version: 1.0
Content-Type: multipart/report; boundary="----=_Part_fba0c199dfcd4d60ae506b37a6320a84"; report-type=notification
To: adam#test.local
Date: Mon, 17 Sep 2018 16:21:45 -0400 (EDT)
From: postmaster#server.local
------=_Part_fba0c199dfcd4d60ae506b37a6320a84
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
BTW, this is the correct notification ... it should be "failed" not "failure".
------=_Part_fba0c199dfcd4d60ae506b37a6320a84
Content-Type: message/notification
Content-Transfer-Encoding: 7bit
Final-Recipient: rfc822; eve#test.local
Original-Message-ID: <06dbfc12-57a5-49bd-84bd-a24bc3ba1e42#fake.cernerasp>
------=_Part_fba0c199dfcd4d60ae506b37a6320a84—
for an example I need to grab what is after Final-Recipient: rfc822; and Original-Message-ID: and store them as variables
I have tried to do something like
if body :raw :contains ["Original-Message-ID:"] {
set "Original-Message-ID" "${0}";
}
but variable set as Original-Message-ID doesn't contain what is after it in the message.
It came up that the body test will definitely not work with match variables:
https://www.rfc-editor.org/rfc/rfc5173#section-6
Wildcard expressions used with "body" are exempt from the side effects
described in [VARIABLES]. That is, they MUST NOT set match variables
(${1}, ${2}...) to the input values corresponding to wildcard
sequences in the matched pattern.
The MIME Sieve extensions will likely provide what is needed
https://www.rfc-editor.org/rfc/rfc5703 or pipe message to a script and process it there, what I have done in my case.
I want custom separators for JSON in Go, equivalent to Python's:
json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
How to specify separator for JSON in Go ?
Clarification:
json.dumps({"key1":"value1","key2":"value2"}, separators=('__','..'))
'{"key2".."value2"__"key1".."value1"}'
There are some valid use cases for this requirement:
http://mens.com?&cyclone_data={"VisitorId":"53905341bd05ae26a9000001","CampaignId":"538f278cbd05ae36c6000001","LandingPageId":"538eac3ebd05ae15fe000001","OfferId":"538f097ebd05ae2a4d000001","SourceId":"538f0e39bd05ae2b9c000002","NetworkId":""}
I need to set this same url in cookie:
tracking_data=http://mens.com?&cyclone_data={"VisitorId":"53905341bd05ae26a9000001","CampaignId":"538f278cbd05ae36c6000001","LandingPageId":"538eac3ebd05ae15fe000001","OfferId":"538f097ebd05ae2a4d000001","SourceId":"538f0e39bd05ae2b9c000002","NetworkId":""}
After setting cookie with http.SetCookie(), Chrome Web Developer tool shows:
Date:Thu, 05 Jun 2014 11:23:46 GMT
Location:http://mens.com?&cyclone_data={"VisitorId":"53905341bd05ae26a9000001","CampaignId":"538f278cbd05ae36c6000001","LandingPageId":"538eac3ebd05ae15fe000001","OfferId":"538f097ebd05ae2a4d000001","SourceId":"538f0e39bd05ae2b9c000002","NetworkId":""}
Set-Cookie:538f278cbd05ae36c6000001=538f278cbd05ae36c6000001; Path=/
Set-Cookie:cyclone-track-url=http://mens.com?&cyclone_data={VisitorId:53905341bd05ae26a9000001CampaignId:538f278cbd05ae36c6000001LandingPageId:538eac3ebd05ae15fe000001OfferId:538f097ebd05ae2a4d000001SourceId:538f0e39bd05ae2b9c000002NetworkId:} Content-Length:378
Content-Type:text/html; charset=utf-8
, and "" are missing.
I am aware of base64. I want users to be able to change VisitorId without any manual encode/decode routine.
Server side app reads and compares this url from Cookie and Browser Referer.
Found this issue:
https://code.google.com/p/go/issues/detail?id=7243
Well, you can't directly, but if what you want is as I'm guessing to remove extra unneeded spaces, you can use json.Compact:
http://golang.org/pkg/encoding/json/#Compact
It just takes your encoded json and removes unnecessary spaces.
You can also try playing with the json.MarshalIndent method, that lets you control indentation separators. But AFAIK, you can't specify non standard separators.
I have a script which processes a shared mailbox and forwards on different email (based on things like the original 'To:' header to various people. It's been working fine for months as a more flexible (and, up till now, reliable) replacement for gmail filters.
The logic is basically
Read each unprocessed message in the inbox
Look up who it was sent 'To:' (e.g. support#domain.com) which is an alias for this gmail inbox
Get a list of the intended recipients (within our organistion)
For each of those, forward the message on
Mark that message as processed, so it doesn't get re-forwarded next time
We've recently received an email which seems to trigger a weird error at the point at which the message is forwarded on. The error is:
Invalid mime type. (line 123, file "Code").
Line 123 of the script reads:
// subject: removes the 'Fwd:' prefix which we don't need, and adds the original target email in square brackets, and sets the replyTo for easy processing
really && msg.forward(rcvr, {subject: msg.getSubject() + " [" + thisTo + "]", replyTo: msg.getFrom()});
Here's a summary of the interesting bits:
really is just a boolean so we can do 'dry runs' for testing
msg is set to the current message being processed, and is of type GmailMessage
rcvr is a string with the recipient's email address
thisTo is a string holding the original 'To:' header
The message in question is a multi-part mime email with plain text and html in it. It displays just fine in gmail, and I can forward it on from there without problems. So, could anyone shed light on what the script engine is complaining about? The raw email looks like this (with some bits taken out for privacy) in case that helps pin it down:
Delivered-To: XXXX#YYYY.com
Received: by 10.182.155.73 with SMTP id vj5sqase239pbc;
Wed, 26 Jun 2013 08:50:48 -0700 (PDT)
X-Received: by 10.68.166.5 with SMTP id zc5m425et238pbb.16.1372261847795;
Wed, 26 Jun 2013 08:50:47 -0700 (PDT)
Return-Path: <support#ZZZZ.com>
Received: from mail.ZZZZ.com (mail.ZZZZ.com. [64.78.193.232])
by mx.google.com with ESMTP id vj5sqwefwe239pbc.316.2013.06.26.08.50.47
for <multiple recipients>;
Wed, 26 Jun 2013 08:50:47 -0700 (PDT)
Received-SPF: pass (google.com: domain of support#ZZZZ.com designates aa.bb.cc.dd as permitted sender) client-ip=aa.bb.cc.dd;
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of support#ZZZZ.com designates aa.bb.cc.dd as permitted sender) smtp.mail=support#ZZZZ.com
Date: Wed, 26 Jun 2013 09:50:46 -0600
To: AAAA#YYYY.com, XXXX#YYYY.com
From: ZZZZ <support#ZZZZ.com>
Reply-to: ZZZZ <support#ZZZZ.com>
Subject: ZZZZ (Order #00412744)
Message-ID: <805b426783f23fec38ddafb002ce40b#admin.ZZZZ.com>
X-Priority: 3
X-Mailer: PHPMailer 5.1 (phpmailer.sourceforge.net)
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="b1_805b426783ff4601fbe72afb002ce40b"
--b1_805b426783ff4601fbe72afb002ce40b
Content-Type: text/plain; charset = "utf-8"
Content-Transfer-Encoding: 7bit
Hello Andy,
[SNIP]
and destroy all copies of the original message. Thank You
--b1_805b426783ff4601fbe72afb002ce40b
Content-Type: text/html; charset = "utf-8"
Content-Transfer-Encoding: 7bit
<html>
<head>
</head>
[SNIP]
and destroy all copies of the original message. Thank You</span></font></p></td></tr></tbody></table></body>
</html>
--b1_805b426783ff4601fbe72afb002ce40b--