How do a format my json data and/or change my function so that it gets stored as columns in Azure table storage?
I am sending a json string to the IoT hub:
{"ts":"2017-03-31T02:14:36.426Z","timeToConnect":"78","batLevel":"83.52","vbat":"3.94"}
I run the sample function (in the Azure Function App module) to transfer the data from the IoT hub into my storage account:
'use strict';
// This function is triggered each time a message is revieved in the IoTHub.
// The message payload is persisted in an Azure Storage Table
var moment = require('moment');
module.exports = function (context, iotHubMessage) {
context.log('Message received: ' + JSON.stringify(iotHubMessage));
context.bindings.deviceData = {
"partitionKey": moment.utc().format('YYYYMMDD'),
"rowKey": moment.utc().format('hhmmss') + process.hrtime()[1] + '',
"message": JSON.stringify(iotHubMessage)
};
context.done();
};
But in my storage table, it shows up as a single string rather than getting split into columns (as seen in the storage explorer.
How do I get it into columns for ts, timeToConnect, batLevel, and vbat?
In case anyone is looking for a solution in c#:
private static async Task ProcessMessage(string message, DateTime enqueuedTime)
{
var deviceData = JsonConvert.DeserializeObject<JObject>(message);
var dynamicTableEntity = new DynamicTableEntity();
dynamicTableEntity.RowKey = enqueuedTime.ToString("yyyy-MM-dd HH:mm:ss.fff");
foreach (KeyValuePair<string, JToken> keyValuePair in deviceData)
{
if (keyValuePair.Key.Equals("MyPartitionKey"))
{
dynamicTableEntity.PartitionKey = keyValuePair.Value.ToString();
}
else if (keyValuePair.Key.Equals("Timestamp")) // if you are using a parameter "Timestamp" it has to be stored in a column named differently because the column "Timestamp" will automatically be filled when adding a line to table storage
{
dynamicTableEntity.Properties.Add("MyTimestamp", EntityProperty.CreateEntityPropertyFromObject(keyValuePair.Value));
}
else
{
dynamicTableEntity.Properties.Add(keyValuePair.Key, EntityProperty.CreateEntityPropertyFromObject(keyValuePair.Value));
}
}
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("myStorageConnectionString");
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("myTableName");
table.CreateIfNotExists();
var tableOperation = TableOperation.Insert(dynamicTableEntity);
await table.ExecuteAsync(tableOperation);
}
How do I get it into columns for ts, timeToConnect, batLevel, and
vbat?
To get these attributes as separate columns in table, you would need to defalte the object and store them separately (currently you are just converting the entire object into string and storing that string).
Please try the following code:
module.exports = function (context, iotHubMessage) {
context.log('Message received: ' + JSON.stringify(iotHubMessage));
var deviceData = {
"partitionKey": moment.utc().format('YYYYMMDD'),
"rowKey": moment.utc().format('hhmmss') + process.hrtime()[1] + '',
};
Object.keys(iotHubMessage).forEach(function(key) {
deviceData[key] = iotHubMessage[key];
});
context.bindings.deviceData = deviceData;
context.done();
};
Please note that I have not tried to execute this code so it may contain some errors.
Related
I am trying to insert array in my firebase collection from cloud function. I need to have multiple lines in one document so for each line i am inserting an array. Please check my attached screenshot where you can see line0 , same way i need to have Line1,Line2,Line3..,Line n in the same document.
for line0 i am passing array from code like below and its working fine.
admin.firestore().collection("qbContestWinners").add(
{
'cmpientryid': context.params.processId,
'qbid': '',
'qbsyncdate': '',
'qbsyncstatus': 'pending',
'Line0':
{
id: "0",
description: 'PRIZE AMOUNT',
amount: 1000,
accountrefid: contestresultData.qbcontestid,
accountrefname: contestresultData.qbcontestname,
contestresultId: context.params.processId,
},
})
when i am looping through data i am getting from another table , i am not able to generate proper JSON to insert.
below is how i am looping and creating JSON after getting data from another table.
i = 1;
admin.firestore().collection("results").where('cid', '==', 'LKRRk2XXXXXXXX')
.orderBy("rank", "asc").get().then(snapshots =>
{
snapshots.forEach(doc =>
{
const contestresultId = doc.id;
const prizeAmount = doc.data().prizeamt;
const userId = doc.data().userid;
const lineNum = "Line" + i;
console.log("new line numner is: ", lineNum);
console.log(`lineNum? ${lineNum}`);
const linetxt = "Line" + String(i);
const insertData = "{"+linetxt +
":{id:'" + i +
"', description: 'PRIZE AMOUNT'"+
", amount:" + prizeAmount + "," +
"accountrefid:"+ contestresultData.qbcontestid +","+
"accountrefname:'" +contestresultData.qbcontestname +"',"+
"contestresultId:'" + contestresultId +"'," +
"},}"
const finalInsert = JSON.stringify(insertData);
const finalJSON = JSON.parse(finalInsert);
admin.firestore().collection("qbContestWinners").doc(mainID).set(
finalInsert.toJSON(),
{
merge: true
});
i= i+1;
});
});
using this code i am getting error
finalInsert.toJSON is not a function
Actually, the Line0 field is a map and not an Array, see this doc for more details.
So, if you want to create similar fields (Line1, Line2, ...), you simply need to pass a JavaScript Object to the set() method, as follows:
snapshots.forEach(doc => {
const contestresultId = doc.id;
const prizeAmount = doc.data().prizeamt;
const userId = doc.data().userid;
const lineNum = "Line" + i;
console.log("new line numner is: ", lineNum);
console.log(`lineNum? ${lineNum}`);
const lineObj = {
id: i,
description: 'PRIZE AMOUNT',
accountrefid: contestresultData.qbcontestid, //Not sure if you have defined contestresultData somewhere...
//...
}
const dataObj = {};
dataObj["Line" + i] = lineObj // See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Property_accessors
admin.firestore().collection("qbContestWinners").doc(mainID).set(dataObj, {merge: true});
i= i+1;
});
HOWEVER, note that you must return a promise that resolves when all the asynchronous work in your Cloud Function is complete (i.e. call to the Firestore set() method).
This is explained in the official Firebase video series, watch in particular the three videos titled "Learn JavaScript Promises".
Since you are calling several times the set() method in a forEach loop, you need to use Promise.all() in order to return a Promise when all these parallel calls to the set() method are completed.
The following should do the trick:
let i = 1;
return admin.firestore().collection("results") // <-- See the return here
.where('cid', '==', 'LKRRk2XXXXXXXX')
.orderBy("rank", "asc").get()
.then(snapshots => {
const promises = [];
snapshots.forEach(doc => {
const contestresultId = doc.id;
const prizeAmount = doc.data().prizeamt;
const userId = doc.data().userid;
const lineNum = "Line" + i;
const lineObj = {
id: i,
description: 'PRIZE AMOUNT',
accountrefid: contestresultData.qbcontestid,
//...
}
const dataObj = {};
dataObj[lineNum] = lineObj;
promises.push(admin.firestore().collection("qbContestWinners").doc(mainID).set(dataObj, {merge: true}));
i= i+1;
});
return Promise.all(promises) // <-- See the return here
});
A last remark: if mainID keeps the same value in the snapshots.forEach loop, you may adopt a totally different approach, consisting in building a JavaScript object with several LineXX properties and call the set() method only once. Since you didn't share the entire code of your Cloud Function it is impossible to say if this approach should be used or not.
first to the error
You stringify and parse a string. The problem here seems to be the order. You have to parse a "String" and to stringify an "Object". The result won't have a toJSON Method as well, but u can just stringify the Object to get a json.
the second thing
Why do you use a string to create your object? You shouldn't. Just use an object.
the third thing
You should not use Objects as Arrays. Not even in firebase.
Just use arrays. Example:
[Line0Object, Line1Object, ...]
Hint: If your array can work as its own collection. Just use a SubCollection. This might fit your needs.
I'm in the process of learning the Forge platform. I'm currently using an example (Jigsawify) written by Kean Walmsley because it most accurately describes my goals. I'm running into an issue of getting my file to download from an Azure Storage Account to Forge. The error I receive is "The value for one of the HTTP headers is not in the correct format." My question is how does someone go about troubleshooting HTTP protocol when writing, in this case, a workitem in code? I can put in a breakpoint to view the workitem, but I'm not versed enough to understand where the flaw is in the HTTP header, or even where to find it. Is there a specific property of the workitem I should be looking at? If I could find the HTTP statement, I could test it, but I don't where I should find it.
Or am I just completely off base?
Anyway here's the code. It's a modified version of what Kean wrote:
static void SubmitWorkItem(Activity activity)
{
Console.WriteLine("Submitting workitem...");
CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(Microsoft.Azure.CloudConfigurationManager.GetSetting("StorageConnectionString"));
StorageCredentials crd = storageAccount.Credentials;
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
CloudFileShare ShareRef = fileClient.GetShareReference("000scrub");
CloudFileDirectory rootDir = ShareRef.GetRootDirectoryReference();
CloudFile Fileshare = rootDir.GetFileReference("3359fort.dwg");
// Create a workitem
var wi = new WorkItem()
{
Id = "", // Must be set to empty
Arguments = new Arguments(),
ActivityId = activity.Id
};
if (Fileshare.Exists())
{
wi.Arguments.InputArguments.Add(new Argument()
{
Name = "HostDwg", // Must match the input parameter in activity
Resource = Fileshare.Uri.ToString(),
StorageProvider = StorageProvider.Generic // Generic HTTP download (vs A360)
});
}
wi.Arguments.OutputArguments.Add(new Argument()
{
Name = "Results", // Must match the output parameter in activity
StorageProvider = StorageProvider.Generic, // Generic HTTP upload (vs A360)
HttpVerb = HttpVerbType.POST, // Use HTTP POST when delivering result
Resource = null, // Use storage provided by AutoCAD.IO
ResourceKind = ResourceKind.ZipPackage // Upload as zip to output dir
});
container.AddToWorkItems(wi);
container.SaveChanges();
// Polling loop
do
{
Console.WriteLine("Sleeping for 2 sec...");
System.Threading.Thread.Sleep(2000);
container.LoadProperty(wi, "Status"); // HTTP request is made here
Console.WriteLine("WorkItem status: {0}", wi.Status);
}
while (
wi.Status == ExecutionStatus.Pending ||
wi.Status == ExecutionStatus.InProgress
);
// Re-query the service so that we can look at the details provided
// by the service
container.MergeOption =
Microsoft.OData.Client.MergeOption.OverwriteChanges;
wi = container.WorkItems.ByKey(wi.Id).GetValue();
// Resource property of the output argument "Results" will have
// the output url
var url =
wi.Arguments.OutputArguments.First(
a => a.Name == "Results"
).Resource;
if (url != null)
DownloadToDocs(url, "SGA.zip");
// Download the status report
url = wi.StatusDetails.Report;
if (url != null)
DownloadToDocs(url, "SGA-Report.txt");
}
Any help is appreciated,
Chuck
Azure requires that you specify the x-ms-blob-type header when you upload to a presigned URL. See https://github.com/Autodesk-Forge/design.automation-.net-input.output.sample/blob/master/Program.cs#L167
So, I was able to figure out how to download my file from Azure to Forge using Albert's suggestion of moving to a blob service. Here's the code:
static void SubmitWorkItem(Activity activity)
{
Console.WriteLine("Submitting workitem...");
CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(Microsoft.Azure.CloudConfigurationManager.GetSetting("StorageConnectionString"));
CloudBlobClient BlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = BlobClient.GetContainerReference("000scrub");
CloudBlockBlob blockBlob = cloudBlobContainer.GetBlockBlobReference("3359fort.dwg");
// Create a workitem
var wi = new WorkItem()
{
Id = "", // Must be set to empty
Arguments = new Arguments(),
ActivityId = activity.Id
};
if (blockBlob.Exists())
{
wi.Arguments.InputArguments.Add(new Argument()
{
Name = "HostDwg", // Must match the input parameter in activity
Resource = blockBlob.Uri.ToString(),
StorageProvider = StorageProvider.Generic, // Generic HTTP download (vs A360)
Headers = new System.Collections.ObjectModel.ObservableCollection<Header>()
{
new Header() { Name = "x-ms-blob-type", Value = "BlockBlob" } // This is required for Azure.
}
});
}
wi.Arguments.OutputArguments.Add(new Argument()
{
Name = "Results", // Must match the output parameter in activity
StorageProvider = StorageProvider.Generic, // Generic HTTP upload (vs A360)
HttpVerb = HttpVerbType.POST, // Use HTTP POST when delivering result
Resource = null, // Use storage provided by AutoCAD.IO
ResourceKind = ResourceKind.ZipPackage, // Upload as zip to output dir
});
container.AddToWorkItems(wi);
container.SaveChanges();
// Polling loop
do
{
Console.WriteLine("Sleeping for 2 sec...");
System.Threading.Thread.Sleep(2000);
container.LoadProperty(wi, "Status"); // HTTP request is made here
Console.WriteLine("WorkItem status: {0}", wi.Status);
}
while (
wi.Status == ExecutionStatus.Pending ||
wi.Status == ExecutionStatus.InProgress
);
// Re-query the service so that we can look at the details provided
// by the service
container.MergeOption =
Microsoft.OData.Client.MergeOption.OverwriteChanges;
wi = container.WorkItems.ByKey(wi.Id).GetValue();
// Resource property of the output argument "Results" will have
// the output url
var url =
wi.Arguments.OutputArguments.First(
a => a.Name == "Results"
).Resource;
if (url != null)
DownloadToDocs(url, "SGA.zip");
// Download the status report
url = wi.StatusDetails.Report;
if (url != null)
DownloadToDocs(url, "SGA-Report.txt");
}
What isn't complete is the result section. The ZIP has nothing in it, but hey, baby steps right?
Thanks Albert.
-Chuck
After saving DateTime in controller, I pass it back to the View but when it's displayed in view the value became /Date(1545062400000)/.
I already checked in the controller while in process if the data was changed but it did not since I'm just passing the viewmodel in the view.
[HttpPost]
public ActionResult UpdateHeader(RecordViewModel recordViewModel)
{
var ResultMessage = "";
if (ModelState.IsValid)
{
Record record = (from c in context.Records
where c.RecordId == recordViewModel.RecordId
select c).FirstOrDefault();
if (record == null)
{
return new HttpStatusCodeResult(HttpStatusCode.BadRequest);
}
record.Description = recordViewModel.Description;
record.Date = recordViewModel.Date;
record.Remarks = recordViewModel.Remarks;
try
{
context.SaveChanges();
ResultMessage = "Record successfully updated.";
}
catch (Exception ex)
{
ModelState.AddModelError("", ErrorHelper.GetInnerException(ex));
ResultMessage = "Error: " + ErrorHelper.GetInnerException(ex);
}
}
var result = new { Model = recordViewModel, ResultMessage = ResultMessage };
return Json(result, JsonRequestBehavior.AllowGet);
}
Angular
self.submitEdit = function () {
var updateRecordHeader = RecordServices.updateRecordHeader(self.header)
.then(function successCallback(response) {
self.header = response.data.Model;
self.header.ResultMessage = response.data.ResultMessage;
}, function errorCallback(response) { toastr.error(response.statusText); });
}
The /Date(1545062400000)/ is known as date ticks using UNIX timestamp format since RFC7519 "epoch" (January 01, 1970), which cannot be directly consumed as JS Date object without converting it first. The reason behind usage of the ticks is JSON format doesn't have specific representation for DateTime struct when serialized to plain strings (see this reference).
You can create a custom function to convert ticks into JS Date object:
function toJSDate(value) {
var regex = /Date\(([^)]+)\)/;
var results = regex.exec(value);
var date = new Date(parseFloat(results[1])); // or parseInt
return date;
}
Or even simpler without regex by picking numeric values directly like this:
function toJSDate(value) {
var date = new Date(parseFloat(value.substring(6))); // or parseInt
return date;
}
Then use that function for ticks-to-date conversion:
// example
var header = response.data.Model;
var dateObject = toJSDate(header.Date);
// assign date object afterwards
Note that you may need to create another object structure which resembles response.data.Model but using server-side Date property with JS date object.
As an alternative you may create a getter-only string property which uses ToString() to convert DateTime value into desired string representation, then use it inside JS.
Side note:
Avoid using viewmodel property name which exactly matches built-in JS function names & objects (i.e. Date) for clarity.
Related issues:
How do I format a Microsoft JSON date?
Converting .NET DateTime to JSON
I am building a bot in Gupshup with Api.ai integration. I have an agent in Api.ai with several intents and each of them linked through contexts(input & output contexts). When I use the following code to call Api.ai, the first intent is called and I get the reply. However when the second message is given, the bot takes it as a completely new message, without identifying its relation with first.
How can I solve this issue? Kindly help
function MessageHandler(context, event) {
// var nlpToken = "xxxxxxxxxxxxxxxxxxxxxxx";//Your API.ai token
// context.sendResponse(JSON.stringify(event));
sendMessageToApiAi({
message : event.message,
sessionId : new Date().getTime() +'api',
nlpToken : "3626fe2d46b64cf8a9c0d3bee99a9sb3",
callback : function(res){
//Sample response from apiai here.
context.sendResponse(JSON.parse(res).result.fulfillment.speech);
}
},context)
}
function sendMessageToApiAi(options,botcontext) {
var message = options.message; // Mandatory
var sessionId = options.sessionId || ""; // optinal
var callback = options.callback;
if (!(callback && typeof callback == 'function')) {
return botcontext.sendResponse("ERROR : type of options.callback should be function and its Mandatory");
}
var nlpToken = options.nlpToken;
if (!nlpToken) {
if (!botcontext.simpledb.botleveldata.config || !botcontext.simpledb.botleveldata.config.nlpToken) {
return botcontext.sendResponse("ERROR : token not set. Please set Api.ai Token to options.nlpToken or context.simpledb.botleveldata.config.nlpToken");
} else {
nlpToken = botcontext.simpledb.botleveldata.config.nlpToken;
}
}
var query = '?v=20150910&query='+ encodeURIComponent(message) +'&sessionId='+sessionId+'&timezone=Asia/Calcutta&lang=en '
var apiurl = "https://api.api.ai/api/query"+query;
var headers = { "Authorization": "Bearer " + nlpToken};
botcontext.simplehttp.makeGet(apiurl, headers, function(context, event) {
if (event.getresp) {
callback(event.getresp);
} else {
callback({})
}
});
}
/** Functions declared below are required **/
function EventHandler(context, event) {
if (!context.simpledb.botleveldata.numinstance)
context.simpledb.botleveldata.numinstance = 0;
numinstances = parseInt(context.simpledb.botleveldata.numinstance) + 1;
context.simpledb.botleveldata.numinstance = numinstances;
context.sendResponse("Thanks for adding me. You are:" + numinstances);
}
function HttpResponseHandler(context, event) {
// if(event.geturl === "http://ip-api.com/json")
context.sendResponse(event.getresp);
}
function DbGetHandler(context, event) {
context.sendResponse("testdbput keyword was last get by:" + event.dbval);
}
function DbPutHandler(context, event) {
context.sendResponse("testdbput keyword was last put by:" + event.dbval);
}
The sessionId has to be fixed for a user. There are two ways you can do this in the Gupshup bot code -
Use the unique userID which is sent to the bot for every user.
To get this value you can use -
event.senderobj.channelid
But this value is dependent on how different messaging channels provides it and api.ai has a limit of 36 characters.
Sample code -
function MessageHandler(context, event) {
sendMessageToApiAi({
message : event.message,
sessionId : event.senderobj.channelid,
nlpToken : "3626fe2d46b64cf8a9c0d3bee99a9sb3",
callback : function(res){
//Sample response from apiai here.
context.sendResponse(JSON.parse(res).result.fulfillment.speech);
}
},context)
}
Generate a unique sessionId for each user and store it in the database to utilise it. In the below sample , I am storing the sessionId at roomleveldata which is the default persistance provided by Gupshup, to know more check this guide.
Sample code -
function MessageHandler(context, event) {
sendMessageToApiAi({
message : event.message,
sessionId : sessionId(context),
nlpToken : "84c813598fb34dc5b1f3e1c695e49811",
callback : function(res){
//Sample response from apiai here.
context.sendResponse(JSON.stringify(res));
}
},context)
}
function sessionId(context){
var userSession = context.simpledb.roomleveldata.sessionID;
if(!userSession){
userSession = new Date().getTime() +'api';
context.simpledb.roomleveldata.sessionID = userSession;
return userSession;
}else{
return userSession;
}
}
Remember that sessionId should not exceed 36 characters.
Suresh,
It seems you generate new session id for every request:
new Date().getTime() +'api'
But if you want to make contexts work it must be one fixed value for all requests belonging to one user. For example, you could use some global variable for it.
I have a requirement wherein I have get the document from couchbase.
Following in the Map function that I am using for the same -
function (doc, meta) {
if (meta.type == "json" && doc!=null) {
emit(doc);
}
}
There is no reduce function. Also following is my java code to get the document -
List<URI> hosts = Arrays.asList(
new URI("http://<some DNS with port>/pools")
);
// Name of the Bucket to connect to
String bucket = "Test-Sessions";
// Password of the bucket (empty) string if none
String password = "";
//System.setProperty("viewmode", "development");
// Connect to the Cluster
CouchbaseClient client = new CouchbaseClient(hosts, bucket, password);
String designDoc = "sessions";
String viewName = "by_test";
View view = client.getView(designDoc, viewName);
Query query = new Query();
query.setIncludeDocs(true);
query.setKey(String.valueOf(122));
ViewResponse result = client.query(view, query);
Object object = null;
for(ViewRow row : result) {
if(null != row) {
object = row.getDocument();
}// deal with the document/data
}
System.out.println("Object" + object);
And the data that I have in couchbase is key - "122" and value - "true". But for some reason , I do not get any rows in the ViewResponse. What is going wrong can anyone help?
I don't understand what you are trying to achieve here, you are using a view to get a document by it's key? Key == 122? Why can't you just do client.get(122) ?
If you just need a list of all the keys in your bucket (of which you can use to pull back all documents via include docs) then make your function like so:
function (doc, meta) {
if (meta.type == "json") {
emit();
}
}
The key of the document is always emitted as an ID (viewRow.getId()). You don't need to emit the document, try to emit as little data as possible to keep view sizes small.
If you are needing to manipulate all the documents in your bucket be careful as the size grows, perhaps you'd need to look at pagination to cycle through the results. http://tugdualgrall.blogspot.com.es/
Also once you have the ViewResponse loop over it like so:
for(ViewRow row : result) {
row.getDocument(); // deal with the document/data
}
You don't need to be doing checks for null on the rows.