I'm trying to write a function to download a file stored in a persistent bucket and I'm having some problems decoding the result.
I'm following the guide here to try and download the object shown here:
(int) 3 => object(stdClass) {
bucketKey => 'my-persistent-bucket'
objectKey => '11--test.dwg'
objectId => 'urn:adsk.objects:os.object:my-persistent-bucket/11--test.dwg'
sha1 => '477085439a60779064d91fd1971d53c77c7a163a'
size => (int) 188600
location => 'https://developer.api.autodesk.com/oss/v2/buckets/my-persistent-bucket/objects/11--test.dwg'
}
Using the following cURL function
$ch = curl_init();
$headers = [
"Authorization: Bearer " . $token->token,
"Accept: application/octet-stream"
];
$url = 'https://developer.api.autodesk.com/oss/v2/buckets/'.$this->persistent.'/objects/'.rawurlencode($file->object_key);
curl_setopt_array($ch, [
CURLOPT_HTTPHEADER => $headers,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => $url
]);
$response = curl_exec($ch) ;
$http_header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE) ;
$http_header = substr($response, 0, $http_header_size) ;
$http_body = substr($response, $http_header_size) ;
$response_info = curl_getinfo($ch) ;
curl_close($ch) ;
curl_getinfo($ch); looks like everything has gone fine:
'url' => 'https://developer.api.autodesk.com/oss/v2/buckets/my-persistent-bucket/objects/11--test.dwg',
'content_type' => 'application/x-www-form-urlencoded',
'http_code' => (int) 200,
'header_size' => (int) 474,
'request_size' => (int) 199,
'filetime' => (int) -1,
'ssl_verify_result' => (int) 0,
'redirect_count' => (int) 0,
'total_time' => (float) 1.261261,
'namelookup_time' => (float) 0.029048,
'connect_time' => (float) 0.057444,
'pretransfer_time' => (float) 0.119675,
'size_upload' => (float) 0,
'size_download' => (float) 188600,
'speed_download' => (float) 149532,
'speed_upload' => (float) 0,
'download_content_length' => (float) 188600,
'upload_content_length' => (float) 0,
'starttransfer_time' => (float) 0.902231,
'redirect_time' => (float) 0,
'redirect_url' => '',
'primary_ip' => '52.210.137.76',
'certinfo' => [],
'primary_port' => (int) 443,
'local_ip' => '10.0.2.15',
'local_port' => (int) 50564
$http_body = '%C8B%BB%8B%A6%12%03Z%7D%29%E7%27%1F%5D%D4%CB%FC%DA%15G%3B%13%0D%89%0A%1C%DB%AE2%2C%9AP%EE%60x6%FD%92I2%F6%DE%7DI%DC%A0O%14%F2%84%9Ed%D0k%C40%B7%3E%3B%A1%22%...
The response is always what looks like a url encoded string but no matter how I try to decode it, I can't manage to get a working file, so far I've tried:
curl_unescape()
urldecode()
rawurldecode()
and none of these give me a useable file. It's worth noting that I can download a file from A360 no problem, but I've not managed to get one out of a Forge bucket.
Any ideas on what I'm doing wrong would be great.
Thanks
The bucketKey should match this regular expression '^[-_.a-z0-9]{3,128}$', and the objectKey should be URL encoded. However, the keys used to upload the file/object must match the ones used to download. Which means that if you choose to encode characters one way, they should be encoded the same way later.
I also noticed you did not specify how to encode the download. If you do not specify to download as octet, HTTP will use the default which is UTF8 text characters vs binary. You did specify CURLOPT_RETURNTRANSFER which will convert to binary, but you are missing the header for the server part.
The code below works fine for me, assuming bucketKey and objectKey were URL encoded with rawurlencode(); as well as when the file/object was uploaded with PUT.
```
$ch =curl_init () ;
$headers =[
"Authorization: Bearer {$token->token}",
"Accept: application/octet-stream",
"Content-Type: application/json"
] ;
$url ="https://developer.api.autodesk.com/oss/v2/buckets/{$bucket_key}/objects/{$object_key}" ;
curl_setopt_array($ch, [
CURLOPT_HTTPHEADER => $headers,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => $url
]) ;
$response =curl_exec ($ch) ;
$http_header_size =curl_getinfo ($curl, CURLINFO_HEADER_SIZE) ;
$http_header =substr ($response, 0, $http_header_size) ;
$http_body =substr ($response, $http_header_size) ;
$response_info =curl_getinfo ($ch) ;
curl_close ($ch) ;
file_put_contents ('11--test.dwg', $http_body) ;
```
As you can see I split the response into 2 parts. The headers and the octet-stream. The file download is in the 2nd part and is the HTTP body response. The response contains both headers and body, hence you need to split this on your end.
Hope that helps,
Related
I am using the following code to get the best selling products and send the result in JSON. But I am unable to encode the result of wc_get_products().
$best_selling_args = array(
'meta_key' => 'total_sales',
'order' => 'DESC',
'orderby' => 'meta_value_num'
);
$products_posts = wc_get_products( $best_selling_args );
// var_dump( $products_posts );
echo wp_json_encode( $products_posts );
I have received arguments in my mason handler which looks in the following format:
$data = {
'cacheParams' => 0,
'requests' => {
'locationId' => 1,
'uniqueId' => [
'ABC',
'DEF',
'XYZ'
]
}
};
I am able to access the requests by using $data['requests']. How do I access the values stored in requests, i.e. locationId and uniqueId ? I need to use these values to form another JSON in the following way:
my $input = {
stateID => 44,
locationId => requests.locationId,
uniqueId => requests.uniqueId
.
.
.
}
The $data['requests'] object should be an hash in your way. So you can access to the keys like in the following:
$data['requests']->{'locationId'}
$data['requests']->{'uniqueId'}
or
$requests = $data['requests']
$locationId = $requests->{'locationId'}
$uniqueId = $requests->{'uniqueId'}
Can someone explain to me how I can trigger Celery tasks through Logstash?
Is it possible?
If I try to do that in PHP through the 'php-amqplib' library it works fine: (without using Logstash)
$connection = new AMQPStreamConnection(
'rabbitmq.local',
5672,
'guest',
'guest'
);
$channel = $connection->channel();
$channel->queue_declare(
'celery',
false,
true,
false,
false
);
$taskId = rand(1000, 10000);
$props = array(
'content_type' => 'application/json',
'content_encoding' => 'utf-8',
);
$body = array(
'task' => 'process_next_task',
'lang' => 'py',
'args' => array('ktest' => 'vtest'),
'kwargs' => array('ktest' => 'vtest'),
'origin' => '#'.'mytest',
'id' => $taskId,
);
$msg = new AMQPMessage(json_encode($body), $props);
$channel->basic_publish($msg, 'celery', 'celery');
According to the Celery docs:
http://docs.celeryproject.org/en/latest/internals/protocol.html
I'm trying to send the request in the json format, this is my Logstash filter:
ruby
{
remove_field => ['headers', '#timestamp', '#version', 'host', 'type']
code => "
event.set('properties',
{
:content_type => 'application/json',
:content_encoding => 'utf-8'
})
"
}
And Celery answer is:
[2017-05-05 14:35:09,090: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!
{content_type:None content_encoding:None delivery_info:{'exchange': 'celery', 'routing_key': 'celery', 'redelivered': False, 'consumer_tag': 'None4', 'delivery_tag': 66} headers={}}
Basically, Celery is not able to decode my message format or better... I'm not able to set the request in the JSON format :)
It's driving me crazy, thank you in advance for any clues :)
Forgot it, this is my output plugin in Logstash
rabbitmq
{
key => "celery"
exchange => "celery"
exchange_type => "direct"
user => "${RABBITMQ_USER}"
password => "${RABBITMQ_PASSWORD}"
host => "${RABBITMQ_HOST}"
port => "${RABBITMQ_PORT}"
durable => true
persistent => true
codec => json
}
From the information provided in this question, you can't.
When you're playing with the event in the ruby filter, you're actually playing with what will be put in the body of the message, while you'd like to set the rabbitmq headers and properties of your message.
Till that functionality has been tackled, I do not think you'll be able to achieve it unless of course you implement it yourself. After all, the plugin is available on github.
As Olivier said, right now is not possible but I've created a pull request to the official project.
https://github.com/logstash-plugins/logstash-output-rabbitmq/pull/59
If you are looking for a working version take a look to my clone:
https://github.com/useless-stuff/logstash-output-rabbitmq
You should be seriously scared about that code :)
I'm completely far away to be a Ruby developer
But it works :)
I'd like to translate a perl web site in several languages. I search for and tried many ideas, but I think the best one is to save all translations inside the mySQL database. But I get a problem...
When a sentence extracted from the database contains a variable (scalar), it prints on the web page as a scalar:
You have $number new messages.
Is there a proper way to reassign $number its actual value ?
Thank you for your help !
You can use printf format strings in your database and pass in values to that.
printf allows you to specify the position of the argument so only have know what position with the list of parameters "$number" is.
For example
#!/usr/bin/perl
use strict;
my $Details = {
'Name' => 'Mr Satch',
'Age' => '31',
'LocationEn' => 'England',
'LocationFr' => 'Angleterre',
'NewMessages' => 20,
'OldMessages' => 120,
};
my $English = q(
Hello %1$s, I see you are %2$s years old and from %3$s
How are you today?
You have %5$i new messages and %6$i old messages
Have a nice day
);
my $French = q{
Bonjour %1$s, je vous vois d'%4$s et âgés de %2$s ans.
Comment allez-vous aujourd'hui?
Vous avez %5$i nouveaux messages et %6$i anciens messages.
Bonne journée.
};
printf($English, #$Details{qw/Name Age LocationEn LocationFr NewMessages OldMessages/});
printf($French, #$Details{qw/Name Age LocationEn LocationFr NewMessages OldMessages/});
This would be a nightmare to maintain so an alternative might be to include an argument list in the database:
#!/usr/bin/perl
use strict;
sub fetch_row {
return {
'Format' => 'You have %i new messages and %i old messages' . "\n",
'Arguments' => 'NewMessages OldMessages',
}
}
sub PrintMessage {
my ($info,$row) = #_;
printf($row->{'Format'},#$info{split(/ +/,$row->{'Arguments'})});
}
my $Details = {
'Name' => 'Mr Satch',
'Age' => '31',
'LocationEn' => 'England',
'LocationFr' => 'Angleterre',
'NewMessages' => 20,
'OldMessages' => 120,
};
my $row = fetch_row();
PrintMessage($Details,$row)
I am trying to parse the JSON written # http://a0.awsstatic.com/pricing/1/ec2/sles-od.min.js
Here is a quick snippet from above link:
{vers:0.01,config:{rate:"perhr",valueColumns:["vCPU","ECU","memoryGiB","storageGB","sles"],currencies:["USD"],regions:[{region:"us-east",instanceTypes:[{type:"generalCurrentGen",sizes:[{size:"t2.micro",vCPU:"1",ECU:"variable",
...
...
...
...
Please visit the aforementioned link to see the complete JSON.
As seen above, none of the keys of above JSON have Double Quotes around them.
This leads to malformed JSON string and my JSON parser is failing at it. I also tried putting this JSON in http://www.jsoneditoronline.org/ and it fails as well.
Now, this is the same link which is used by Amazon to display various prices of their EC2 instance. So I think I am missing something here. My Googling led me to believe that above thing is not JSON and is instead JSONP.. I don't understand what is that.
Could you help me understand how to parse this JSON. BTW, I am doing this work in perl using JSON Module.
Some background:
Amazon Web Services does not have an API to get Pricing info programmatically. Hence I am parsing these links which is what amazon is doing while displaying pricing information here. Besides, I am not from programming space and perl is all I know.
Like you said JSONP or "JSON with padding" can't be parsed by json parser because it is not json (it is a different format). But it is actually a json with the prefix (padding)
The padding is typically the name of a callback function that wraps json.
In this case, its default callback names 'callback' and we can do a bit hackiest way by using Regular Expression to capture json that is wrapped by 'callback()' like this
s/callback\((.*)\);$/$1/s;
Also, if you would like to use JSON library, you can enable allow_barekey which means you don't need those quotes around those keys.
Below is my working code. I use LWP::Simple to get the content for the given and Data::Dump to print the isolated data structure.
use strict;
use warnings;
use LWP::Simple;
use JSON;
my $jsonp = get("http://a0.awsstatic.com/pricing/1/ec2/sles-od.min.js")
or die "Couldn't get url";
( my $json = $jsonp ) =~ s/callback\((.*)\);$/$1/s; #grap the json from $jsonp and store in $json variable
my $hash = JSON->new->allow_barekey->decode ( $json );
use Data::Dump;
dd $hash;
Outputs:
{
config => {
currencies => ["USD"],
rate => "perhr",
regions => [
{
instanceTypes => [
{
sizes => [
{
ECU => "variable",
memoryGiB => 1,
size => "t2.micro",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.023 } }],
vCPU => 1,
},
{
ECU => "variable",
memoryGiB => 2,
size => "t2.small",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.056 } }],
vCPU => 1,
},
{
ECU => "variable",
memoryGiB => 4,
size => "t2.medium",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.152 } }],
vCPU => 2,
},
{
ECU => 3,
memoryGiB => 3.75,
size => "m3.medium",
storageGB => "1 x 4 SSD",
valueColumns => [{ name => "os", prices => { USD => "0.170" } }],
vCPU => 1,
},
....
As said in comments above, it is not JSON so it can't be parsed by JSON parser... But for an quick & (very)dirty work, you can try the JSON::DWIW module.
The next code:
use 5.014;
use warnings;
use WWW::Mechanize;
use Data::Dump;
use JSON::DWIW;
my $mech = WWW::Mechanize->new();
my $jsonstr = $mech->get('http://a0.awsstatic.com/pricing/1/ec2/sles-od.min.js')->content;
($jsonstr) = $jsonstr =~ /callback\((.*)\)/s;
my $json_obj = JSON::DWIW->new;
my $data = $json_obj->from_json( $jsonstr );
dd $data;
prints a structure what maybe is what you want, e.g.:
{
config => {
currencies => ["USD"],
rate => "perhr",
regions => [
{
instanceTypes => [
{
sizes => [
{
ECU => "variable",
memoryGiB => 1,
size => "t2.micro",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.023 } }],
vCPU => 1,
},
{