OrientDB ETL Throws exception on loading CSV file - csv

I am trying to load a simple CSV file to OrientDB and it always throws this exception.
CSV File Content
id, name, role
1, Sarath, Architect
2, Anoop, Project Manager
3, Nazeem, Lead Developer
4, Rini, Senior Developer
5, Shine, iOS Developer
6, Vishnu, iOS Developer
json config file
{
"source": { "file": { "path": "./dev.csv" } },
"extractor": { "row": {} },
"transformers": [
{
"csv": {}
},
{ "vertex": { "class": "Person" } }
],
"loader": {
"orientdb": {
"dbURL": "remote:localhost/dev",
"dbType": "graph",
"dbUser": "root",
"dbPassword": "root",
"dbAutoCreate": true,
"classes": [
{"name": "Person", "extends": "V"}
], "indexes": [
{"class":"Person", "fields":["id:integer"], "type":"UNIQUE" }
]
}
}
}
All the time it shows this exception. I tried different CSV options but it did not work. Looks like the file reading itself throwing exception.
» oetl posts.json sarat#Saraths-MacBook-Air
OrientDB etl v.2.0.1 (build #BUILD#) www.orientechnologies.com
Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on loading config file: posts.json
at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:151)
Platform Details
OS X 10.10.2
java version "1.8.0_25"
OrientDB - v.2.0.1

The stack trace gives you the cause of the problem (you only showed the first, not interesting lines in your example).
Check:
The server is running (you specified remote:localhost/dev, so the server must be running on the default port at the same machine).
The database exists and has "root" / "root" as credentials (note that mostly "admin" "admin" is used)
The file dev.csv is present in the current directory
Tried out your example and works without modification.

I had the same problem, and it was due to one of the directories in the path to the JSON config file having a space in it. The oetl script doesn't handle this well and thus can't find the script.
#rmuller: The OP showed the entire traceback. That's all you get in this case.

Related

Can my bot server retrieve any information in the app manifest?

I am adding a new static tab to an existing teams app.
One of the new features is that the chat will publish a deep link to the static tab. This is creating a dilemna when I am contemplating deployment.
Obviously I can’t put my entire application up here, but we can assume a situation like this:
The manifest I have in production that is in the App Store looks like this:
{
"$schema": "https://developer.microsoft.com/en-us/json-schemas/teams/v1.9/MicrosoftTeams.schema.json",
"manifestVersion": "1.9",
"version": "2.0.1",
"id": "23232322-7676-4848-5555-44444444444",
"packageName": "xxxxx",
"developer": {
"name": "xxxxx",
"websiteUrl": "https://www.xxxx.com/",
"privacyUrl": "https://www.xxxx.com/legal/privacy",
"termsOfUseUrl": "https://www.xxxxx.com/legal/privacy"
},
"icons": {
"color": "color.png",
"outline": "outline.png"
},
"name": {
"short": "abra",
"full": "abra cadabra"
},
"description": {
"short": "short",
"full": "full"
},
"accentColor": "#FFFFFF",
"bots": [
{
"botId": "11111111-2222-3333-4444-555555555555",
"scopes": [
"personal"
],
"commandLists": [
{
"scopes": [
"personal"
],
"commands": [
{
"title": "Test",
"description": "Test"
}
]
}
],
"supportsFiles": false,
"isNotificationOnly": false
}
],
"permissions": [
"identity",
"messageTeamMembers"
],
"validDomains": [
"token.botframework.com",
"teams.microsoft.com",
"*.ngroc.io"
]
}
And in my bot in response to the “Test” command I return a simple text message, i.e.
return await step.prompt('textPrompt', { prompt: 'In the next version we will have a test tab' })
Now in development, I have added the following section to the manifest (after “bots”)
"staticTabs": [
{
"entityId": "conversations",
"scopes": [
"personal"
]
},
{
"entityId": "testtab”,
"contentBotId": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
"name": “Go To Test Tab",
"scopes": [ "personal" ]
},
{
"entityId": "about",
"scopes": [
"personal"
]
}
],
And I have updated the value of the version from 2.0.1 to 2.1.0
Now, in response to the “Test” message I am returning an adaptiveCard that looks like this:
{
type: 'AdaptiveCard',
version: '1.3',
body: [],
actions: [
{
type: 'Action.OpenUrl',
title: 'Go To Test Tab',
url: ‘https://teams.microsoft.com/l/entity/23232322-7676-4848-5555-44444444444/testtab’
}
]
}
This works very nicely and when I press the “go to Test Tab” it opens up the test tab as desired.
My problem is with deploying.
If I submit my new manifest to the App Store, without updating my production server, those testing the new version will continue to get the message “In the next version we will have a test tab” and be missing the link to the new tab.
If I update my production server now, before the update in the App Store, my current users will get the adaptive card with the button that will fail if they press it.
Clearly, neither of these options are good.
Ideally, my response should be something like:
If tabExists() {
Return {… the adaptive card }
Else {
return await step.prompt('textPrompt', { prompt: 'In the next version we will have a ‘test tab })
}
However, I don't know how to write "tabExists()" - I was thinking maybe there was some way to access the version number from the bot manifest, or the list of static tabs, and use that information, but I am happy for alternative suggestions.
We can get the manifest version using below Graph API or using SDK method:
https://learn.microsoft.com/en-us/graph/api/appcatalogs-list-teamsapps?view=graph-rest-1.0&tabs=http#example-4-list-applications-with-a-given-id-and-return-the-submission-review-state
So, before sending the Adaptive Card or message, you can put a custom condition to check the manifest version and send the message accordingly.
You can check the manifest version as explained by Chetan, i tried the same and figured its actually not the best idea.
It always should be best practice to have a "production" and "test" server or something like that.
Having different manifest versions point to the same server and running checks which version you're on will more than likely lead to issues sooner or later.
I sometimes even have 3 different instances running (prod, staging, test) to try various builds and things.
You can check that in your code but imo its not advisable to do so.

Angular 6 - Unexpected token in JSON at position 0

My Angular 5 project was working without issues, just after having updated it to version 6, it stopped building using ng build due to the next:
ERROR in ./src/app/assets/i18/en.json Module parse failed: Unexpected
token in JSON at position 0 You may need an appropriate loader to
handle this file type.
here is my json file:
{
"app": {
"Welcome": "Welcome",
"New": "New"
},
"mainMenu": {
"Home": "Home",
"Logout": "Logout"
},
"pageHeader": {
"About": "About",
"Settings": "Settings"
}
}
Most solutions on the web are talking about CopyWebpackPlugin but the project doesn't use any Webpack configuration file.
then, following this link I tried to make the json as an array:
{
"menu":[
"app": {
"Welcome": "Welcome",
"New": "New"
},
"mainMenu": {
"Home": "Home",
"Logout": "Logout"
},
"pageHeader": {
"About": "About",
"Settings": "Settings"
}
]
}
But got the following error, despite the file contains 16 lines.
Unexpected token : in JSON at position 24
Any idea ?
Inspired by #AndrewJuniorHoward, found that while upgrade process, all the json files were encoded to UTF-8-BOM instead of UTF-8, that's why Angular was unable to load them during build.
In Visual Studio code, I just created empty files, pasted in them the content of the old json files and then overwritten them, and all worked perfectly.
Resave the angular.json file as UTF8. There seems to be a recent problem with upgrading to Angular 6 regarding this.
Hope you have resolved the issue but still if you want some minor changes you can try adding "id" to objects in array as below, I Tried this in my CLI project on Angular 6 while performing CURD operation in JSON file.
{
"menu":[
"app": {
"id": 1,
"Welcome": "Welcome",
"New": "New"
},
"mainMenu": {
"id": 2,
"Home": "Home",
"Logout": "Logout"
},
"pageHeader": {
"id": 3,
"About": "About",
"Settings": "Settings"
}
]
}

OStorageException occurs while creating a vertex using the etl process in Orientdb

I am relatively new to OrientDB.
I want to load a couple of CSV files to OrientDb. I loaded one of the file as a vertex and it worked fine, similarly, when I try to load another file the below exception is thrown
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local
storage '../databases/AABbd' with mode=rw
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot
open storage it is acquired by other process
The first file (member-gm.csv) contained values as shown below
MEMBERID,ENROLLBEGINDATE,ENROLLENDDATE,Dental,Drug,MHInpt,MHDN,MHAMB,CDInpt,CDDN,CDAMB,Hospice,Payer,PEFlag,Ind
95000,20141118,20160328,Y,Y,Y,Y,Y,Y,Y,Y,N,MCS,N,A
95001,20150225,20181231,Y,N,Y,Y,Y,Y,Y,Y,N,POS,N,A
95002,20150607,20160620,Y,Y,Y,Y,Y,Y,Y,Y,N,MLI,N,
The Json(Members.json) file which was created for loading the above file is
{
"source": { "file": { "path": "/G:/AAB/member-gm.csv" } },
"extractor": { "csv": { "columns": ["MEMBERID","Gender","DOB","LName","FName","MMidName","SubID","Add1","Add2","City","State","MZip","MPhone","PFirstName","PMidName","PLastName","Race","Ethn","RaceDS","EthnDS","SpokenLang","SpokenLangSource","WrittenLang","WrittenLangSource","OtherLang","OtherSource"],
"columnsOnFirstLine": true } },
"transformers": [
{ "vertex": { "class": "Member" } }
],
"loader": {
"orientdb": {
"dbURL": "plocal:../databases/AABbd",
"dbType": "graph",
"classes": [
{"name": "Member", "extends": "V"}
]
}
}
}
I executed the json file with the command
G:\orientdb-community-importers-2.2.29\bin> oetl.bat /AAB/Members.json
Database AABdb and Vertex Member was created.
The 2nd vertex which I want to create is the payer vertex and the related csv(Payer_category.csv) is as below
ID,Payer_Line,Payer_Name,Input_Payer,Output_payer
1,Medicaid,Medicaid,MMP,MCD
2,Medicare,Medicaid Dual Eligible HMO,MDE,MCR
3,Medicaid,Medicaid Dual Eligible HMO,MDE,MCD
The Related Payer.json file contains
{
"source": { "file": { "path": "/G:/AAB/Payer_category.csv" } },
"extractor": { "csv": { "columns": ["ID","Payer_Line","Payer_Name","Input_Payer","Output_payer"],
"columnsOnFirstLine": true } },
"transformers": [
{ "vertex": { "class": "Payer" } }
],
"loader": {
"orientdb": {
"dbURL": "plocal:../databases/AABbd",
"dbType": "graph",
"classes": [
{"name": "Payer", "extends": "V"}
]
}
}
}
When I execute the
G:\orientdb-community-importers-2.2.29\bin> oetl.bat /AAB/Payer.json
The below error is thrown
G:\orientdb-community-importers-2.2.29\bin>oetl.bat /AAB/Payer.json
OrientDB etl v.2.2.29 (build 9914189f972103907c24377a1567897e68642920)
https://www.orientdb.com
[file] INFO Load from file /G:/AAB/Payer_category.csv
[csv] INFO column types: {Payer_Name=ANY, Payer_Line=ANY, ID=ANY, Input_Payer=AN
Y, Output_payer=ANY}
[orientdb] INFO Opening database 'plocal:../databases/AABbd'...
$ANSI{green {db=AABbd}} Exception `3023DF74` in storage `AABbd`
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local
storage '../databases/AABbd' with mode=rw
DB name="AABbd"
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginat
edStorage.open(OAbstractPaginatedStorage.java:304)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.ope
n(ODatabaseDocumentTx.java:259)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configureDocu
mentDB(OOrientDBLoader.java:400)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configure(OOr
ientDBLoader.java:343)
at com.orientechnologies.orient.etl.OETLProcessor.configureComponent(OET
LProcessor.java:470)
at com.orientechnologies.orient.etl.OETLProcessor.configureLoader(OETLPr
ocessor.java:291)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.ja
va:223)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.ja
va:187)
at com.orientechnologies.orient.etl.OETLProcessor.parseConfigAndParamete
rs(OETLProcessor.java:155)
at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.jav
a:119)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot
open storage it is acquired by other process
DB name="AABbd"
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPagin
atedStorageDirtyFlag.lockFile(OPaginatedStorageDirtyFlag.java:96)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPagin
atedStorageDirtyFlag.open(OPaginatedStorageDirtyFlag.java:129)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocal
PaginatedStorage.preOpenSteps(OLocalPaginatedStorage.java:424)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginat
edStorage.open(OAbstractPaginatedStorage.java:258)
... 9 more
Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigur
ationException: Error on creating ETL processor
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.ja
va:239)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.ja
va:187)
at com.orientechnologies.orient.etl.OETLProcessor.parseConfigAndParamete
rs(OETLProcessor.java:155)
at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.jav
a:119)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot
open local storage '../databases/AABbd' with mode=rw
DB name="AABbd"
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginat
edStorage.open(OAbstractPaginatedStorage.java:304)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.ope
n(ODatabaseDocumentTx.java:259)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configureDocu
mentDB(OOrientDBLoader.java:400)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configure(OOr
ientDBLoader.java:343)
at com.orientechnologies.orient.etl.OETLProcessor.configureComponent(OET
LProcessor.java:470)
at com.orientechnologies.orient.etl.OETLProcessor.configureLoader(OETLPr
ocessor.java:291)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.ja
va:223)
... 3 more
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot
open storage it is acquired by other process
DB name="AABbd"
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPagin
atedStorageDirtyFlag.lockFile(OPaginatedStorageDirtyFlag.java:96)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPagin
atedStorageDirtyFlag.open(OPaginatedStorageDirtyFlag.java:129)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocal
PaginatedStorage.preOpenSteps(OLocalPaginatedStorage.java:424)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginat
edStorage.open(OAbstractPaginatedStorage.java:258)
... 9 more
All this is directly done on the server.
I tried using the dbUsername and dbPassword properties as well, the result was the same
---------Edit----------------------------
When I try to create a new DB with second json file(Payer.json), a new db with the vertex Payer is created, its just that in an already existing db a new vertex is not getting created.
Any help will be much appreciated. Thanks
is it normal that the files have different name member-gm.csv member-en.csv ??
I try the case and it works fine, by the way, this error:
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '../databases/AABbd' with mode=rw
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot open storage it is acquired by other process
it might be caused the server is up and running and you are doing the import with plocal or your server is down but you're connected by console....
hope it helps
Regards

How to set the application log in an Azure webapp using Azure resource manager

Does anyone know how to set the below diagnostic settings on an Azure Web App using Azure Resource Manager (Powershell or JSON).
Using .json I can only find these settings
"requestTracingEnabled": true, /* Failed request tracing, aka 'freb' */
"httpLoggingEnabled": true, /* IIS logs (aka Web server logging) */
"logsDirectorySizeLimit": 40, /* 40 MB limit for IIS logs */
"detailedErrorLoggingEnabled": true, /* Detailed error messages */
Which turns on web server logging to filesystem but NOT Application Logging or to blob storage.
With Powershell this command only seems to work with ASM as it does not find Non-classic storage accounts given to it
Enable-AzureWebsiteApplicationDiagnostic
Any help would be appreciated. We are currently using Azure Powershell 0.9.8
Regards
The Azure Resource Manager (ARM) template json section for configuring Application Log (Blob) and Web Server Logging (Storage) based on your screenshot above is as shown below:
{
"apiVersion": "2015-08-01",
"name": "logs",
"type": "config",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites', parameters('siteName'))]"
],
"properties": {
"applicationLogs": {
"azureBlobStorage": {
"level": "Information",
"sasUrl": "<Your Azure Blob Storage Account SAS Url>",
"retentionInDays": null
}
},
"httpLogs": {
"azureBlobStorage": {
"sasUrl": "<Your Azure Blob Storage Account SAS Url>",
"retentionInDays": null,
"enabled": true
}
},
"failedRequestsTracing": {
"enabled": true
},
"detailedErrorMessages": {
"enabled": true
}
}
}
References: AzureWebsitesSamples/ARMTemplates/WebAppManyFeatures.json
Hope this answer your question and help solving your issue.
Please let me know if you need further help or clarifications.
If you browse an existing webapp in Resource Explorer you'll find a config/logs section that looks something like this:
{
"id": "/subscriptions/.../config/logs",
"name": "logs",
"type": "Microsoft.Web/sites/config",
"location": "North Central US",
"properties": {
"applicationLogs": {
"fileSystem": {
"level": "Off"
},
"azureBlobStorage": {
"level": "Information",
"sasUrl": "...",
"retentionInDays": 14
}
},
...
}
I believe you can use this format in your json template to configure the logging. (This section would be a sibling to the config/web section which contains the settings mentioned in the question.)
Note though that the config/logs section is not described in the System.Web schema, and so I'd imagine is not currently supported by MS at this time. I'm pretty sure I've tried it and seen it work though.
I didn't find how to set it in the template at the first provision. But it turned out to be easy after resources were provisioned. Please see answer at https://stackoverflow.com/a/51617949/511144 for the actual script to do it that uses Set-AzureRmResource

How to configure LumberJack(LogStash-forwarder) on windows

I've installed ELK on my ubuntu server using this manual,
Now i want to index some log files from a windows server so I installed a logstash forwarder (LumberJack), but I can't get it to run.
this is the logstash-forwarder.conf file :
{
"network": {
"servers": [ "http://XX.XX.XX.XX:5000" ],
"ssl key": "D:/lumberjack/pki/tls/certs/logstash-forwarder.crt",
"ssl ca": "D:/lumberjack/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15,
},
"files": [
{
"paths": [
#single paths are fine
"D:/bea12/Oracle/Middleware/domains/Google/servers/RT1/logs/AppLogs/RT1_APP_9_0.log",
#globs are fine too, they will be periodically evaluated
#to see if any new files match the wildcard.
"/var/logauth.log"
],
]
}
}
and this is the Error I get when i'm trying to run the "lumberjack.exe" ,
That I created with go-build:
2015/04/30 18:17:39.052033 Failed unmarshalling json: invalid character '}' looking for beginning of object key string
2015/04/30 18:17:39.052033 Could not load config file d:\lumberjack\logstash-forwarder.conf: invalid character '}' looking for beginning of object key string
Can anyone please tell me what am I doing wrong?
By the way this is the command I'm using to run the forwarder:
lumberjack.exe -config="d:\lumberjack\logstash-forwarder.conf"
Ok.
So the problem was in the configuration file, There were 2 unnecesary commas and no need for the http:\ at the start:
{
"network": {
"servers": [ "XX.XX.XX.XX:5000" ],
"ssl key": "D:/lumberjack/pki/tls/certs/logstash-forwarder.key",
"ssl ca": "D:/lumberjack/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
#single paths are fine
"D:/bea12/Oracle/Middleware/domains/google/servers/RT1/logs/AppLogs/RT1_APP_9_0.log",
#globs are fine too, they will be periodically evaluated
#to see if any new files match the wildcard.
"/var/logauth.log"
]
}
]
}
This is my suggested configuration file for LumberJack on windows.