I am trying to convert a simple json file to avro using avro tools (1.7.7).
The command I've been running
java -jar ~/Downloads/avro-tools-1.7.7.jar fromjson
--schema-file src/main/avro/twitter.avsc tweet.json > tweet.avro
on this schema
{
"type": "record",
"name": "tweet",
"namespace": "co.feeb.avro",
"fields": [
{
"name": "username",
"type": "string",
"doc": "screen name of the user on twitter.com"
},
{
"name": "text",
"type": "string",
"doc": "the content of the user's message"
},
{
"name": "timestamp",
"type": "long",
"doc": "unix epoch time in seconds"
}
],
"doc": "Schema for twitter messages"
}
I see this exception after running this command:
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1436)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1337)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.avro.tool.Util.openFromFS(Util.java:88)
at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:82)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.lang.NumberFormatException: For input string: "810d:340:1770::1"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at com.sun.jndi.dns.DnsClient.<init>(DnsClient.java:127)
at com.sun.jndi.dns.Resolver.<init>(Resolver.java:61)
at com.sun.jndi.dns.DnsContext.getResolver(DnsContext.java:573)
at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:434)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:235)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:141)
at com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103)
at sun.security.krb5.KrbServiceLocator.getKerberosService(KrbServiceLocator.java:85)
at sun.security.krb5.Config.checkRealm(Config.java:1120)
at sun.security.krb5.Config.getRealmFromDNS(Config.java:1093)
at sun.security.krb5.Config.getDefaultRealm(Config.java:987)
at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:81)
Trying to prefer IPv4 over IPv6 using -Djava.net.preferIPv4Stack=true didn't help. (I am running Mac OSX 10.10.3 and Java 1.8.0_25-b17).
Oh snap ... I've solved it myself right after posting this. My local router added an IPv6 nameserver to my local machine.
Changing the assigned nameserver manually to Google's 8.8.8.8 fixed the issue.
Related
I am creating an ARM template that creates databases(array) on an Azure SQL server and I am trying to output the ADO.NET connection strings and save them in the Key vault. I have 3 connection strings (SQL authentication, AD password Authentication and AD integrated authentication). I am getting the outputs by giving them individually like below:
"outputs": {
"SQLAuthConnectionString": {
"type": "object",
"value": {
"type": "string",
"value": "[concat('Server=tcp:',reference(parameters('serverName')).fullyQualifiedDomainName,',1433;Initial Catalog=',parameters('dbnameName'),';Persist Security Info=False;User ID=',reference(parameters('serverName')).administratorLogin,';Password=',reference(parameters('serverName')).administratorLoginPassword,';MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;')]"
}
},
"ADPasswordConnectionString": {
"type": "object",
"value": {
"type": "string",
"value": "[concat('Server=tcp:',reference(parameters('serverName')).fullyQualifiedDomainName,',1433;Initial Catalog=',parameters('dbnameName'),';Persist Security Info=False;User ID=',reference(parameters('serverName')).administratorLogin,';Password=',reference(parameters('serverName')).administratorLoginPassword,';MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Authentication="Active Directory Password"]"
}
},
"ADIntegratedConnectionString": {
"type": "object",
"value": {
"type": "string",
"value": "[concat('Server=tcp:',parameters('serverName').fullyQualifiedDomainName,,,1433;Initial Catalog=',parameters('databaseName'),';Persist Security Info=False;User ID=',reference(parameters('serverName'))]"
However checking to see if we can add conditions in outputs (using a parameter ex: parameter('authentication_method') or any other way). Any inputs appreciated as this is my first arm template.
As mentioned in the Azure Documentation, you can use conditions in the output block of a ARM template when you've conditionally deployed a resource.
You can refer this document, for simple example of conditional output.
We are upgrading our Data pipeline version from 3.3.2 to 5.8, so those bootstrap actions on old AMI release have changed to be setup using configuration and specifying them under classification / property definition.
So my Json looks like below
{
"enableDebugging": "true",
"taskInstanceBidPrice": "1",
"terminateAfter": "2 Hours",
"name": "ExportCluster",
"taskInstanceType": "m1.xlarge",
"schedule": {
"ref": "Default"
},
"emrLogUri": "s3://emr-script-logs/",
"coreInstanceType": "m1.xlarge",
"coreInstanceCount": "1",
"taskInstanceCount": "4",
"masterInstanceType": "m3.xlarge",
"keyPair": "XXXX",
"applications": ["hadoop","hive", "tez"],
"subnetId": "XXXXX",
"logUri": "s3://pipelinedata/XXX",
"releaseLabel": "emr-5.8.0",
"type": "EmrCluster",
"id": "EmrClusterWithNewEMRVersion",
"configuration": [
{ "ref": "configureEmrHiveSite" }
]
},
{
"myComment": "This object configures hive-site xml.",
"name": "HiveSite Configuration",
"type": "HiveSiteConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},
{
"myComment": "This object sets a hive-site configuration
property value.",
"name":"hive-exec-compress-output",
"type": "Property",
"id": "hive-exec-compress-output",
"key": "hive.exec.compress.output",
"value": "true"
}
],
"parameters": []
With the above Json file it gets loaded into Data Pipeline but throws an error saying
Object:HiveSite Configuration
ERROR: 'HiveSiteConfiguration'
Object:ExportCluster
ERROR: 'configuration' values must be of type 'null'. Found values of type 'null'
I am not sure what this really means and could you please let me know if i am specifying this correctly which i think i am according to http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html
The below block should have the name as "EMR Configuration" only then its recognized correctly by the AWS Data pipeline and the Hive-site.xml is being set accordingly.
{
"myComment": "This object configures hive-site xml.",
"name": "EMR Configuration",
"type": "EmrConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},
I'm trying to write a powershell script that creates a new streamAnalytics job in my azure portal account, with input source as iot-hub and output source as blob storage account.
To do so, I'm using AzureRM command new-streamAnalyticsJob, and json files.
my problem is: I have not seen any documentation or example for json file where the inputs source is iot-hub. only event-hub.
what are the parameters I need to give in the json file? can anyone display an example for json file with input source to streamAnalytics job as Iot-hub?
I got the answer eventually: the required field I had to add to the inputs Oliver posted earlier here is:
"endpoint":"messages/events"
I added it under Datasource Properties section, and it works fine!
Thanks Oliver
To come back on the error message you are seeing, to add to Olivier's sample you need a Property named endpoint which corresponds to the endpoint in IoT Hub. If you are looking for Telemetry messages this will be:
"endpoint": "messages/events"
This can be found in the schema for Azure ARM: https://github.com/Azure/azure-rest-api-specs/blob/current/specification/streamanalytics/resource-manager/Microsoft.StreamAnalytics/2016-03-01/examples/Input_Create_Stream_IoTHub_Avro.json
So to complete Olivier's example, when using API version '':
"Inputs": [
{
"Name": "Hub",
"Properties": {
"DataSource": {
"Properties": {
"consumerGroupName": "[variables('asaConsumerGroup')]",
"iotHubNamespace": "[parameters('iotHubName')]",
"sharedAccessPolicyKey": "[listkeys(variables('iotHubKeyResource'), variables('iotHubVersion')).primaryKey]",
"sharedAccessPolicyName": "[variables('iotHubKeyName')]",
"endpoint": "messages/events"
},
"Type": "Microsoft.Devices/IotHubs"
},
"Serialization": {
"Properties": {
"Encoding": "UTF8"
},
"Type": "Json"
},
"Type": "Stream"
}
}
],
That'd look like the following for the inputs part of the ASA resource:
"Inputs": [
{
"Name": "IoTHubStream",
"Properties": {
"DataSource": {
"Properties": {
"consumerGroupName": "[variables('CGName')]",
"iotHubNamespace": "[variables('iotHubName')]",
"sharedAccessPolicyKey": "[listkeys(variables('iotHubKeyResource'), variables('iotHubVersion')).primaryKey]",
"sharedAccessPolicyName": "[variables('iotHubKeyName')]"
},
"Type": "Microsoft.Devices/IotHubs"
},
"Serialization": {
"Properties": {
"Encoding": "UTF8"
},
"Type": "Json"
},
"Type": "Stream"
}
}
]
I am trying to move an S3 bucket from one account (A) to another (B).
I have succeeded with that operation and remove the bucket from account A.
I am trying to move the new bucket from account B to another bucket on account B, but learning that beside the bucket itself I have no access to the files.
After much fighting with s3 cli and its permissions I checked s3api commands and found out that the files (surprise surprise) still holds the old ownership.
I am trying now to change it, but came to a stand still with the put-bucket-acl, the JSON file isn't working for s3api command.
I tried running the command in debug , but didn't make too much out of it.
Anybody knows what to do ?
Maybe a better way to solve this issue ?
what I did so far:
the command:
aws s3api put-bucket-acl --bucket my-bucket --cli-input-json file://1.json
(Same with put-object-acl)
1.json file:
"Grantee": {
"DisplayName": "account_B",
"EmailAddress": "user#mail.com",
"ID": "111111hughalphnumericnumber22222",
"Type": "CanonicalUser",
"Permission": "FULL_CONTROL"
}
The errors I get :
Unknown parameter in input: "Grantee", must be one of: ACL,
AccessControlPolicy, Bucket, ContentMD5, GrantFullControl, GrantRead,
GrantReadACP, GrantWrite, GrantWriteACP Unknown parameter in input:
"Permission", must be one of: ACL, AccessControlPolicy, Bucket,
ContentMD5, GrantFullControl, GrantRead, GrantReadACP, GrantWrite,
GrantWriteACP
UPDATE:
AssumeRole between the 2 accounts doesn't work in my case.
cli (s3cmd,s3api) GUI (MCSTools,bucketexplorer), ACL using headers,body (Postman) did not help as well..
I'm connecting AWS support and hoping for the best.
I'll update when I have a solution.
So, AWS support came to the rescue... I'm leaving this for others to see, so they won't have to waste 2 days like I did trying to figure what the hell went wrong...
aws s3api get-object-acl --bucket <bucket_on_B> --key <Key_on_B_Owned_by_A> --profile IAM_User_A > A_to_B.json
apply the outcome of:
aws s3api get-bucket-acl --bucket <Bucket_on_B> --profile IAM_User_B
onto the json file that was created, and then run
aws s3api put-object-acl --bucket <Bucket_on_B> --key <Key_on_B_Owned_by_A> --access-control-policy file://A_to_B.json --profile IAM_User_A
Your JSON is wrong. According to the documentation for the put-bucket-acl option you can generate valid JSON template ('skeleton') using --generate-cli-skeleton. For example:
aws s3api put-bucket-acl --bucket BUCKETNAME --generate-cli-skeleton
And here is the output:
{
"ACL": "",
"AccessControlPolicy": {
"Grants": [
{
"Grantee": {
"DisplayName": "",
"EmailAddress": "",
"ID": "",
"Type": "",
"URI": ""
},
"Permission": ""
}
],
"Owner": {
"DisplayName": "",
"ID": ""
}
},
"Bucket": "",
"ContentMD5": "",
"GrantFullControl": "",
"GrantRead": "",
"GrantReadACP": "",
"GrantWrite": "",
"GrantWriteACP": ""
}
For anyone who's still looking to do this - OP probably looked at the right aws doc but overlooked the right command. I'm just glad I got to right command because of this stackoverflow page :)
https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-acl.html
^^ The json syntax with example is present there and instead of --cli-input-json , use --access-control-policy
{
"Grants": [
{
"Grantee": {
"DisplayName": "string",
"EmailAddress": "string",
"ID": "string",
"Type": "CanonicalUser"|"AmazonCustomerByEmail"|"Group",
"URI": "string"
},
"Permission": "FULL_CONTROL"|"WRITE"|"WRITE_ACP"|"READ"|"READ_ACP"
}
...
],
"Owner": {
"DisplayName": "string",
"ID": "string"
}
}
I had the policy as a json file and used this command it worked just fine.
aws s3api put-bucket-acl --bucket bucketname --access-control-policy file://yourJson.json
Also one more thing to note is that I wasn't able to add permissions along with existing ones, old acl was being overwritten. So any permission you want to add needs to be in json policy file along with existing policy. It will be easier when you use some command to describe all the ACLs first.
The syntax is the following (with example):
aws s3api put-bucket-acl --bucket bucket_name --access-control-policy file://grant.json
grant.json file:
{
"Grants": [
{
"Grantee": {
"ID": "CANONICAL_ID_TO_GRANT",
"Type": "CanonicalUser"
},
"Permission": "WRITE"
},
{
"Grantee": {
"ID": "CANONICAL_ID_TO_GRANT",
"Type": "CanonicalUser"
},
"Permission": "READ"
}
],
"Owner": {
"DisplayName": "example_owner",
"ID": "CANONICAL_ID_OWNER"
}
}
I'm a newbie to Swagger. I've used the swagger servlet to generate my swagger.json file from our REST API Java classes. The swagger.json file shows swagger 2.0 (I assume this is the 2.0 schema version). There was nothing fancy in the source files, just #Api and a few #ApiOperation annotations.
Then I tried using swagger-codegen-cli (both version 2.1.4 and 2.1.6-SNAPSHOT, the latest) to generate HTML output from the JSON file. I got the following results on both:
reading from dsm_swagger.json
[main] ERROR io.swagger.codegen.DefaultCodegen - unexpected missing property for name suppressed
[main] WARN io.swagger.codegen.DefaultCodegen - skipping invalid property {
"type" : "array"
}
writing file /home/combs/dsm_swagger/./index.html
So I get an output file, but any types that are flagged as lists of objects are not handled correctly. These do appear to be valid 2.0 constructs.
I'm also getting Jackson errors about invalid escape characters because it sees
"pattern": "\d+"
in the file. I can work around the \d by using [0-9], but assume it should be handled as is.
Has anybody seen these particular issues and know if they're either fixed or there is a workaround in swagger-codegen or the source file? Is swagger-codegen actually handling v2.0 specs correctly? Any pointers to up to date info or code would be appreciated!
EDIT:
As mentioned in a comment, by using "#JsonIgnore" and "#JsonProperty" in appropriate places and upgrading to V1.5.6 of swagger-core, I got around the issues with invalid property and type "array" messages. Here's an example of the issue with \d:
"/v1/admins/{adminId}": {
"put": {
"tags": [
"admins"
],
"summary": "Update information about a particular admin, given its ID. The update information is passed in the POST body.",
"description": "Longer notes about what this does",
"operationId": "updateUser",
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"parameters": [
{
"name": "adminId",
"in": "path",
"required": true,
"type": "integer",
"pattern": "\d+",
"format": "int64"
},
{
"in": "body",
"name": "body",
"required": false,
"schema": {
"$ref": "#/definitions/UserUpdateInfo"
}
}
],
"responses": {
"200": {
"description": "successful operation",
"schema": {
"$ref": "#/definitions/UserInfo"
}
}
}
}
},
This is the exact output of swagger-core, and yet swagger-codegen fails with the following:
combs#dcombs-lap:~/dsm_swagger$ gen_file
reading from dsm_swagger.json
reading from dsm_swagger.json
com.fasterxml.jackson.core.JsonParseException: Unrecognized character escape 'd' (code 100)
at [Source: dsm_swagger.json; line: 411, column: 27]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508)
at com.fasterxml.jackson.core.base.ParserMinimalBase._handleUnrecognizedCharacterEscape(ParserMinimalBase.java:485)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeEscaped(UTF8StreamJsonParser.java:2924)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2209)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2165)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:279)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:224)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:262)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:221)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:218)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:218)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:218)
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:62)
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:14)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3066)
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:1861)
at io.swagger.parser.SwaggerCompatConverter.readResourceListing(SwaggerCompatConverter.java:139)
at io.swagger.parser.SwaggerCompatConverter.read(SwaggerCompatConverter.java:74)
at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:73)
at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:317)
at io.swagger.codegen.cmd.Generate.run(Generate.java:186)
at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)
Exception in thread "main" java.lang.RuntimeException: missing swagger input or config!
at io.swagger.codegen.DefaultGenerator.generate(DefaultGenerator.java:89)
at io.swagger.codegen.cmd.Generate.run(Generate.java:188)
at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)
combs#dcombs-lap:~/dsm_swagger$