I would like to be able to update my data table like this :
Each one of the 608 update represent a date :
So basically my jobSpec is like this :
var jobSpec = {
configuration: {
load: {
destinationTable: {
projectId: projectId,
datasetId: 'Facebook',
tableId: tableId
},
allowJaggedRows: true,
writeDisposition: 'WRITE_TRUNCATE',
schema: {
fields: [
{name: 'Page_ID', type: 'STRING'},
{name: 'Post_ID', type: 'STRING'},
{name: 'Post_creation_date', type: 'STRING'},
{name: 'Post_name', type: 'STRING'},
{name: 'Post_message', type: 'STRING'}
]
}
}
}
};
and here is my job :
BigQuery.Jobs.insert(jobSpec, projectId, data);
I tried to remplace 'WRITE_TRUNCATE' by 'WRITE_APPEND' but it's merging all my update. I would like to keep track of them as I shown in my screenshot.
Thanks !
Not sure if I fully understood your question but in order to create tables like the ga_sessions all you have to do is to create tables with the same prefix and just change some identification for them.
For instance, if you go to your BigQuery WebUI and create a table called "test_1" and then create another one just like the first but named "test_2" you will see the same result as in ga_sessions (but this time you will see test_(2)).
If you want to use the API, you'd have to do something like:
BigQuery.Jobs.insert(jobSpec, projectId, data, table_id='test_1');
BigQuery.Jobs.insert(jobSpec, projectId, data, table_id='test_2');
So it's not the "write_append" nor the "write_truncate" that you should be changing but rather the table's name.
This type of partitioning is more "manual" and you are the one resposible for creating the different tables.
BigQuery offers a more automatic option as well, which is a partitioned table. This type of table is a bit different from the ga_sessions in the sense that all you will have is just one table. And all data inserted in this table say in day 28 April will be allocated automatically to this timestamp. If next day you insert more data, it automatically is allocated to the timestamp 29 April and so on.
Later on to query your data you can use the _PARTITIONTIME to select only the desired timestamp.
It's a matter of identifying which one makes more sense to you in your project.
Related
I think I've done enough research on this subject and I've only got a headache.
Here is what I have done and understood: I have restructured my MySQL database so that I will keep my user's data in different tables, I am using foreign keys. Until now I only concluded that foreign keys are only used for consistency and control and they do not automatize or do anything else (for example, to insert data about the same user in two tables I need to use two separate insert statements and the foreign key will not help to make this different or automatic in some way).
Fine. Here is what I want to do: I want to use Sequelize to insert, update and retrieve data altogether from all the related tables at once and I have absolutely no idea on how to do that. For example, if a user registers, I want to be able to insert the data in the table "A" containing some user information and in the same task insert in the table B some other data (like the user's settings in the dedicated table or whatever). Same with retrievals, I want to be able to get an object (or array) with all the related data from different tables fitting in the criteria I want to find by.
Sequelize documentation covers the things in a way that every thing depends on the previous one, and Sequelize is pretty bloated with a lot of stuff I do not need. I do not want to use .sync(). I do not want to use migrations. I have the structure of my database created already and I want Sequelize to attach to it.
Is it possible insert and retrieve several rows related at the same time and getting / using a single Sequelize command / object? How?
Again, by "related data" I mean data "linked" by sharing the same foreign key.
Is it possible insert and retrieve several rows related at the same
time and getting / using a single Sequelize command / object? How?
Yes. What you need is eager loading.
Look at the following example
const User = sequelize.define('user', {
username: Sequelize.STRING,
});
const Address = sequelize.define('add', {
address: Sequelize.STRING,
});
const Designation = sequelize.define('designation', {
designation: Sequelize.STRING,
});
User.hasOne(Address);
User.hasMany(Designation);
sequelize.sync({ force: true })
.then(() => User.create({
username: 'test123',
add: {
address: 'this is dummy address'
},
designations: [
{ designation: 'designation1' },
{ designation: 'designation2' },
],
}, { include: [Address, Designation] }))
.then(user => {
User.findAll({
include: [Address, Designation],
}).then((result) => {
console.log(result);
});
});
In console.log, you will get all the data with all its associated models that you want to include in the query
I'm using the sequelize syntax to generate a unique composite with two columns in a table like so:
sequelize.define('Group', {
code: {
type: DataTypes.STRING,
unique: 'GroupCompositeIndex'
},
active: {
type: DataTypes.BOOLEAN,
unique: 'GroupCompositeIndex'
}
});
The goal is to have many inactive versions of this group code and change which is active; only one at a time. I will be changing the active value to null on one record, and to 1 on another. I have never constructed a table to work like this. Will it behave the way I expect by precluding there from ever being 2 groups active with the same code, while permitting which group is active to change?
I am working on a sails applications which contains multiple(>2) tables which I need to join with the help of populate method
e.g.
Category.js model
attributes: {
CategoryID:{
type:"integer",
required:true,
primaryKey:true,
autoIncrement:true
},
SubCategories:{ //REFERING TO SUB-CATEGORY TABLE
collection:'SubCategory',
via:'CategoryID'
},
CategoryName:{
type:"string",
required:true,
maxLength:50
}
}
this is SubCategory.js model.
attributes: {
id:{
type:'integer',
required:true,
primaryKey:true,
autoIncrement:true,
maxLength:10,
columnName:'SubCategoryID'
},
CategoryID:{
model:'Category' //REFERING TO CATEGORY TABLE
},
ProductsOfCategory:{ //REFERING TO PRODUCT TABLE
collection:'Product',
via:'SubCategoryID'
},
SubCategory:{
type:"string",
required:true,
maxLength:50
}
}
and Product.js model
attributes: {
id: {
type: 'integer',
primaryKey: true,
autoIncrement: true,
maxLength: 10,
columnName:'ProductID'
},
SubCategoryID: {
model:'SubCategory'
},
ProductDetails:{
collection:'ProductDetails',
via:'ProductID'
},
ProductName: {
type: "string",
required: true,
maxLength: 50
}
}
and ProductDeatils.js model
attributes: {
id: {
type: 'integer',
primaryKey: true,
autoIncrement: true,
maxLength: 10,
columnName:'ProductDetailID'
},
ProductID:{
model:'Product'
},
Size:{
type:"string",
required:true,
maxLength:10
},
Color:{
type:"string",
required:true,
maxLength:10
}
}
On Populating, I am able to populate the category and sub-category of each category.
Category.find()
.populate('SubCategories')
.exec(function(err, categories){
if (err) {
console.log(err);
return res.json(err);
}
console.log(categories);
res.json(categories);
})
How to populate the all above table in one go such that after final query we get all the above details in one json.
We get join of all above tables
that is category having all sub-categories, sub-category having all products and all product have product details in one json
You ask a great question. There has been massive interest in getting nested populate feature into sails, literally tens of issue requests and PRs etc.
Take a look at some history here:
[FEATURE REQUEST] Recursively populate #308 - i was late to the party, making the request on October 29th 2014 as you'll see in the history.
As far as I know, most conversations eventually converged here (after a couple of years of Sails users requesting the feature):
Deep populate #1052 (the issue remains open as of writing 14 Jan 2016)
It is unclear from the state of that Issue where we are. The history of both links does suggest alternative workarounds others have used.
My hunch is that recursive populate is not supported out of the box.
What I did when using waterline model associations with SailsJS, was work with a package like async.js - use something like waterfall to explicitly populate the child relationships programmatically. You can combine doing this with overriding the default toJSON() of the models you invoke to add their relationships (which you have programmatically populated) to the JSON response. You could equally choose to use the built-in promises to achieve the same thing.
Found this (dated, 2014) SOF Question which offers more information.
Someone, do please correct me here if I have missed this feature addition in a recent Sails or Waterline version - couldn't find anything in the release notes for either project to say this was supported.
I have a query I'm trying to perform based on a one to many relationship.
As an example there is a model called Users and one called Projects.
Users hasMany Projects
Projects have many types which are stored in a type (enum) column. There are 4 different types that potentially a user may have that I want to load. The catch is I want to include the most recent project record (createdAt column) for all networks that potentially will be there. I have not found a way to structure the query for it to work as an include. I have however found a way to do a raw query which does what I want.
I am looking for a way without having to do a raw query. By doing the raw query I have to map the returned results to users I've returned from the other method, or I have to do a simple include and then trim off all the results that are not the most recent. The latter is fine, but I see this getting slower as a user will have many projects and it will keep growing steadily.
This allow serialize a json for anywhere action about a model. Read it, very well
sequelize-virtual-fields
// define models
var Person = sequelize.define('Person', { name: Sequelize.STRING });
var Task = sequelize.define('Task', {
name: Sequelize.STRING,
nameWithPerson: {
type: Sequelize.VIRTUAL,
get: function() { return this.name + ' (' + this.Person.name + ')' }
attributes: [ 'name' ],
include: [ { model: Person, attributes: [ 'name' ] } ],
order: [ ['name'], [ Person, 'name' ] ]
}
});
// define associations
Task.belongsTo(Person);
Person.hasMany(Task);
// activate virtual fields functionality
sequelize.initVirtualFields();
I am trying to load some JSON, in which I store a lot of variables about some 100 anaesthetic drugs for pediatric patients.
The actual values get calculated before from patient's weight, age etc.:
Example:
var propofolInductionTitle = propofolName + ' ' + propofol0PercentConcentration + '- Induktion';
var propofol0InductionDosageMG = (Math.round(kg * 2 * 10) / 10) + ' - ' + (Math.round(kg * 5 * 10) / 10);
I then create my drug as a block of json consisting of the variables I need which are later to be replaced by the calculated values. I specifically try to avoid Strings in the JSON to allow for easier localization to english and french when all variables are defined in the math block.
var hypnotikaJSON = {
"thiopentalTitle": [
{"thiopentalBrandName": ""},
{"vialContentTitle": "thiopentalVialContent"},
{"solutionTitle": "thiopentalSolution"},
{"concentrationTitle": "thiopentalConcentration"},
{"dosageString": "thiopentalDosageString"},
{"atWeight": "thiopentalDosageMG"},
{"thiopentalAtConcentration": "thiopentalDosageML"}
],
"propofolInductionTitle": [
{"propofolInductionBrandName": ""},
{"propofolVialContentTitle": "propofolInductionVialContent"},
{"propofolSolutionTitle": "propofolSolution"},
{"propofolConcentrationTitle": "propofolInductionConcentration"},
{"propofolInductionDosageStringTitle": "propofolInductionDosageString"},
{"atWeight": "propofolInductionDosageMG"},
{"propofolAtInductionConcentration": "propofolInductionDosageML"}
],
"propofolSedationTitle": [
{"propofolSedationBrandName":""},
{"propofolVialContentTitle":"propofolSedationVialContent"},
{"propofolSolutionTitle":"propofolSolution"},
{"propofolConcentrationTitle":"propofolSedationConcentration"},
{"propofolSedationDosageStringTitle":"propofolSedationDosageString"},
{"atWeight":"propofolSedationDosageMG"},
{"propofolAtSedationConcentration":"propofolSedationDosageML"}
],
"laryngealMaskTitle": [
{"laryngealMaskSizeTitle":"laryngealMaskSize"},
{"laryngealMaskCuffSizeTitle":"laryngealMaskCuffSize"},
{"laryngealMaskBiggestETTTitle":"laryngealMaskBiggestETT"},
{"laryngealMaskBronchoscopeSizeTitle":"laryngealMaskBronchoscopeSize"}
]
};
My specific need is that the JSON reader has to give me the key AND value of each object as I need both to populate a template. The reason ist that the fields for the drugs are different in parts. Some have additional routes of administration so I have another key:value pair a different drug doesnt have. Some are given both as bolus and per drip, some arent. So no convenient json structure ist possible.
I found an answer by rdougan here that partly allowed me to do just that:
Model:
Ext.define('my.model.Drug', {
extend: 'Ext.data.Model',
config: {
fields: ['name', 'value']
}
});
Custom Json Reader:
Ext.define('Ext.data.reader.Custom', {
extend: 'Ext.data.reader.Json',
alias: 'reader.custom',
getRoot: function (data) {
if (this.rootAccessor) {
data = this.rootAccessor.call(this, data);
}
var values = [],
name;
for (name in data) {
values.push({
name: name,
value: data[name]
});
}
return values;
}
});
Store:
var store = Ext.create('Ext.data.Store', {
fields: ['name', 'value'],
data: hypnotikaJSON,
autoLoad: true,
proxy: {
type: 'memory',
reader: {
type: 'custom'
}
}
});
Panel:
this.viewport = new Ext.Panel({
fullscreen: true,
layout: 'fit',
items: [{
xtype: 'list',
itemTpl: '<p class="description">{name}</p><p class ="values">{value}</p>',
store: store
}]
});
Unfortunately I'm a physician and no programmer, and after a lot of reading I cant find out to apply this to a nested JSON. The custom reader seems to only go for the first level.
I could do it without a reader, without a store with just a lot of plan html around each entry, that has proven to be very very slow though so I would like to avoid it while updating from Sencha Touch 1.1. and better do it right this time.
Could you please point me to a way to parse this ugly data structure?
Thank you
I don't know much about extending JSON readers, so just guessing, but maybe you are supposed override the 'read' method? Then you can go over the JSON as you wish
Also, if you have control over the JSON you should consider changing it.
Usually, the keys in JSON should be the same throughout all items in the array.
keys are not data, they are metadata.
So, if you do have different properties between different drugs, then something like this might be a solution:
[{
name: 'Propofol 1%',
properties: [
{title: 'induction', value: '22-56g'},
{title: 'Sedation', value: '22'},
etc.
]},
{name: 'nextDrug'}
etc..