I have two Json files that I exported from wordpress that have corresponding ID's I want to combine them into one Json file so I can bring it into website I am building with Gatsby JS. One of the files is the posts.json and the other is postsMeta.json. The post_id in postsMeta corresponds with the ID in Posts
How would I best go about merging the two? Can I run some sort of for loop in js and how would I so? I am on windows is there a json explorer of some sorts that could help me do this.
lastly I would also like to trim out some of the unnecasry fiels such as post_parent in the posts json and something like the meta_key in the postsMeta json.
Ok hopefully this is clear enough, thanks in advance.
Here is an example of the first object corresponding pairs in the two files
posts.json
{"ID":"19","post_author":"2","post_date":"2010-12-31 23:02:04","post_date_gmt":"2010-12-31 23:02:04","post_content":"Harry Potter was not available for the first sitting of the Halloween Picture. I hope everyone had a safe and fun Halloween. Tomorrow is picture retake day, please send back your previous prints if you want retakes. It is also hot lunch. See You tomorrow!","post_title":"Happy Halloween","post_excerpt":"","post_status":"publish","comment_status":"open","ping_status":"open","post_password":"","post_name":"happy-halloween","to_ping":"","pinged":"","post_modified":"2011-01-03 05:26:11","post_modified_gmt":"2011-01-03 05:26:11","post_content_filtered":"","post_parent":"0","guid":"http:\/\/localhost\/mrskitson.ca_wordpress\/?p=19","menu_order":"0","post_type":"post","post_mime_type":"","comment_count":"1"},
postsMeta.json
{"meta_id":"27","post_id":"19","meta_key":"large_preview","meta_value":"http:\/\/www.mrskitson.ca\/wp-content\/uploads\/2010\/12\/halloween.jpg"},
Update:
this is an attempt to solve this problem with the current answer, you can edit the code there.
How would I best go about merging the two?
Is it mandatory for you combine the two JSON files/data?
Because you could just require or load the JSON data from within your script (or even put them in the HTML) and then to get the meta value of a specific meta field/key, this function could do that:
// `single` has no effect if `meta_key` is empty.
function getPostMeta( post_id, meta_key, single ) {
let id = String( post_id ),
pm = [];
postsMeta.map( m => {
let a = ( ! meta_key ) ||
( meta_key === m.meta_key );
if ( a && id === m.post_id ) {
pm.push( m );
}
});
let meta = {},
mk = {};
pm.map( m => {
let k = m.meta_key, v;
if ( undefined === meta[ k ] ) {
meta[ k ] = m.meta_value;
} else {
v = meta[ k ];
if ( undefined === mk[ k ] ) {
meta[ k ] = [ v ];
mk[ k ] = 1;
}
meta[ k ].push( m.meta_value );
m[ k ]++;
}
});
pm = null;
mk = meta_key ? mk[ meta_key ] : null;
if ( mk ) {
return single ?
meta[ meta_key ][0] : // Returns a single meta value.
meta[ meta_key ]; // Returns all the meta values.
}
return meta_key ?
meta[ meta_key ] : // Returns the value of the `meta_key`.
meta; // Or returns all the post's meta data.
}
The data I used for testing: (take note of the postsMeta in the above/getPostMeta() function)
// Array of `post` objects.
const posts = [{"ID":"19","post_author":"2","post_date":"2010-12-31 23:02:04","post_date_gmt":"2010-12-31 23:02:04","post_content":"Harry Potter was not available for the first sitting of the Halloween Picture. I hope everyone had a safe and fun Halloween. Tomorrow is picture retake day, please send back your previous prints if you want retakes. It is also hot lunch. See You tomorrow!","post_title":"Happy Halloween","post_excerpt":"","post_status":"publish","comment_status":"open","ping_status":"open","post_password":"","post_name":"happy-halloween","to_ping":"","pinged":"","post_modified":"2011-01-03 05:26:11","post_modified_gmt":"2011-01-03 05:26:11","post_content_filtered":"","post_parent":"0","guid":"http:\/\/localhost\/mrskitson.ca_wordpress\/?p=19","menu_order":"0","post_type":"post","post_mime_type":"","comment_count":"1"}];
// Array of `meta` objects.
const postsMeta = [{"meta_id":"27","post_id":"19","meta_key":"large_preview","meta_value":"http:\/\/www.mrskitson.ca\/wp-content\/uploads\/2010\/12\/halloween.jpg"},{"meta_id":"28","post_id":"19","meta_key":"many_values","meta_value":"http:\/\/facebook.com"},{"meta_id":"29","post_id":"19","meta_key":"many_values","meta_value":"http:\/\/twitter.com"},{"meta_id":"30","post_id":"19","meta_key":"many_values","meta_value":"http:\/\/linkedin.com"}];
Examples: (see this Fiddle for demo)
// In these examples, we are retrieving the meta value for the post #19 (i.e. ID is 19).
// Retrieve a single value.
// Returns mixed; string, number, etc.
let url = getPostMeta( 19, 'large_preview', true );
console.log( url );
// Retrieve all meta values.
// Always returns an array of values.
let ms = getPostMeta( 19, 'many_values' );
console.log( ms, ms[0] );
// Retrieve all meta data.
// Always returns an object with meta_key => meta_value pairs. I.e. { key => value, ... }
let ma = getPostMeta( 19 );
console.log( ma, ma.large_preview, ma.many_values[0] );
But if you really must combine the JSON data, you can do: (again, see demo on the same Fiddle)
// Here we modify the original `posts` object.
posts.map( p => {
// Add all the post's meta data.
p.meta = getPostMeta( p.ID );
// Delete items you don't want..
delete p.post_parent;
delete p.menu_order;
// delete ...;
});
console.log( JSON.stringify( posts[0].meta ) ); // posts[0].meta = object
console.log( posts[0].post_parent, posts[0].menu_order ); // both are undefined
And then if you want to copy-paste the new/merged JSON data:
JSON.stringify( posts );
But if you actually just want to do something with the post's meta, you can loop through the posts object and do the thing; e.g.:
// Here the original `posts` object is not modified, and that we don't
// (though you can) repeatedly call `getPostMeta()` for the same post.
posts.map( p => {
// Get all the post's meta data.
let meta = getPostMeta( p.ID );
// Do something with `meta`.
console.log( meta.large_preview );
});
console.log( JSON.stringify( posts[0].meta ) ); // posts[0].meta = undefined
console.log( posts[0].post_parent, posts[0].menu_order ); // both still defined
// posts[0].meta wouldn't be undefined if of course posts[0] had a `meta` item,
// which was set in/via WordPress...
If you can do this in js, there's a pretty easy approach using Array#map. If you simplify your question, you're really asking how to add this meta data under each entry in posts, and get only the fields you want.
I'm assuming the posts.json is actually an array (e.g. [{"ID":"19"....).
// Load these server-side, fetch them remotely, copy-paste, etc.
// I'll require them here for simplicity
const posts = require('./posts.json');
const postsMeta = require('./postsMeta.json');
// Build a Map so we can quickly look up the metas by post_id
// Extract what we need by destructuring the args
const metaByPost = postsMeta.reduce((a, {
post_id: id,
meta_value: value,
}) => a.set(id, {
value,
/* anything else you want in here */,
}), new Map());
const mergedPosts = posts.map(post => ({
// Spread in the post
...post,
// Spread in the meta content
...metaByPost.get(post.ID),
// Undefine the props we don't want
post_parent: undefined,
}));
I don't love manually setting stuff to undefined -- I think it's nicer to explicitly say what props you're going to include, instead of loading everything and undefining certain props.
Try this snippet directly in the Chrome DevTools console:
(function(
postsUrl='https://cdn.glitch.com/61300ea6-6cc4-4cb6-a62f-31adc62ea5cc%2Fposts.json?1525386749382',
metaUrl='https://cdn.glitch.com/61300ea6-6cc4-4cb6-a62f-31adc62ea5cc%2Fpostmeta.json?1525386742630'
) {
Promise.all([
fetch(postsUrl).then(r => r.json()),
fetch(metaUrl).then(r => r.json()),
]).then(([postsResponse, metaResponse]) => {
// Inspected the actual JSON response to come up with the data structure
const posts = postsResponse[2].data;
const meta = metaResponse[2].data;
const metaByPostId = meta.reduce((accum, el) => {
accum[el.post_id] = el;
return accum;
}, {});
const transformedPosts = posts.map(post => {
const merged = {
...post,
...(metaByPostId[post.ID] || {}),
};
delete merged.post_parent;
// delete any other fields not wanted in the result
return merged;
});
console.log(transformedPosts);
});
})();
replace URLs accordingly, I used the ones from Glitch example here
as commented, actual data is buried in response[2].data. Use Network tab / Parsed view to see structure
replace console.log with copy, if you want the result copied to clipboard, instead of logged to console
Blunt to the point for your question. We want to:
merge var a = {/*some json*/} into var b = {/*another json*/}
trim fields in var exclusions = ["post_parent","meta_key"]
Merge the JSONS
First, we need to populate a and b.
Your JSONs are parsable into Javascript objects with JSON.parse():
let a = JSON.parse(/*JSON here*/);
let b = JSON.parse(/*JSON here*/);
Because how properties are defined in Javascript, if you define a property again, the second definition will overwrite the first. Your JSONS contain only strings as keys and strings as values, so a shallow copy will suffice. Object.assign() will copy all the properties (field and values) into the first argument and return the final Object. Therefore this will merge a into b, assuming they have different keys, else the values in b will overwrite values in a:
a = Object.assign(a,b);
Otherwise, if they are not disjoint, you have to define some policy on how to join, for example may prioritize one. Below, we keep the values in a instead:
a = Object.assign(b,a);
Since you mentionmed a for loop, the line below does the same as two code lines above and will also allow show you an example on how to write your own custom lambda expression:
Object.keys(a).forEach(k=>b[k]=b[k]?b[k]:a[k]);
Do not wish to touch a and b? Create a third object c.
let c = Object.assign({},a,b)
Lastly (wait until the trim step below is accomplished) JSON.stringify() will convert your merged object back into JSON.
Trim exclusions
Following the third example, we have c merged with all the fields.
First a little hack taken from here:
Object.filter = (obj, predicate) => Object.keys(obj)
.filter( key => predicate(obj[key]))
.reduce( (res, key) => (res[key] = obj[key], res), {} );
Now Objects, just like arrays have a filter prototype, having extended Object prototype. It is not really best practice since this will extend every Object but this function works quite well with respects to the semantics of Javascript and this example serves as an opportunity to keep elegant Javascript styles code:
c = Object.filter(c, key=> !exclusions.includes(key) );
Voit-lá, done.
As for defined Object.filter() it uses Array.filter() and Array.reduce() . Click for reference, for your convenience.
Related
I am using this json server in my Angular app, to create, fetch, and delete posts.
In the following method, I delete a post with a specified id:
deleteConsumer(post: Post): Observable<Post> {
const url = `${this.apiUrl}/${post.id}`;
return this.httpClient.delete<Post>(url);
}
I looked at the .delete code and searched for something like a .deleteall but could not find it. Is there really no such method that would delete everything?
If there really isn't, then my attempt at doing it myself is not paying off, because what I have done is not working:
deleteConsumers(): Observable<Post> {
let i: number = 0;
this.httpClient.get<Post[]>(this.apiUrl).forEach(
() => {
++i;
const url = `${this.apiUrl}/${i}`;
return this.httpClient.delete<Post>(url);
}
);
}
Obviously, this is wrong in terms of return type, but I cannot figure out what to do... How can I modify the first method, so it would go through all the json objects in my db.json file; meaning iterate through all the existing posts and delete them all?
I did encounter this when using json-server with Vue.js and I realized that there was no special function to delete all at once. I had to work around it.
So, for example in your case, I would first map the posts array to get a new array with only the post ids:
const postsIdsArray = this.posts.map((post) => post.id)
Then, assuming you already have a function to delete one post given the id, I would then execute the function for each of the ids in the array:
postsIdsArray.forEach((id) => this.deletePost(id))
Just combine the two lines in one JavaScript function (in this case I used Vue.js):
deleteAllPosts(){
const postsIdsArray = this.posts.map((post) => post.id)
postsIdsArray.forEach((id) => this.deletePost(id))
}
My node.js MySQL query returns a single row wrapped in [RowPacketData] which I can normally access the ID field using results[0].ID.
However, when I store the result in React state (using hooks) it does not work. I can access the result object, but not fields within it.
function MyReactComponent() {
const [dbEntry, setDbEntry] = useState();
useEffect(() => {
const fetchData = async () => {
const result = await queryFunc(`SELECT * FROM table LIMIT 1`);
console.log(result[0]); // <-- Works (shows [RowDataPacket] object)
console.log(result[0].ID); // <-- Works (shows ID)
setDbEntry(result);
};
fetchData();
}, []);
console.log(dbEntry[0]); // <-- Works (shows [RowDataPacket] object)
console.log(dbEntry[0].ID); // <-- TypeError: Cannot read property '0' of undefined
return (
<p>
{dbEntry[0].ID} // <-- How do I render here?
</p>
)
}
What's going on here? I have a feeling React is coercing the result object somehow, but I can't figure it out...
When you need to display data that comes from an async font(API calls for example), it's possible (actually almost certain) that it won't be available by the time the first render occurs, to solve that there is actually a few things you could do:
Placeholder state
You could have a model of what the data will look like described as your initial state, so properties won't be undefined anymore:
const [state, setState] = useState({
data:[
{name: ''}
]
})
Assuming that your data will have this format accessing state.data[0].name won't throw an error. This could be useful in some cases but I personally don't like the approach.
Conditional Render
At each render you should check for a condition and only if satisfied render the piece of code:
return(
<>
<div>Title</div>
{Boolean(state.data.length) && <div>{state.data[0].name}</div>}
</>
)
Suspense
That one is brand new, if you have a component tha't need to perform side effects before render it's content, you should have a fallback content to be displayed while the async action is being perform.
<Suspense fallback={<span>Loading</span>}>
<MYAsyncComponent />
</Suspense>
Given this query here,
let output = [];
const sql = `select * from coredb.account LIMIT ${offset},${limit}`;
let data = await sequelize.query(sql, null, {raw: true, type: sequelize.QueryTypes.SELECT});
data.forEach((item) => {
console.log(item['id'], item.id); // <-- output says "undefined, undefined"
});
the data variable is indeed hydrated with the right row data when using console.log to inspect it.
But, when I try to access the individual properties, they only ever come back as undefined. This TextRow object that Sequelize seems to return the result in doesn't seem to want to let me access then explicit rows.
Just curious what i'm missing here, am I missing an option?
I agree, Sequalize raw queries are not intuitive. You don't need the null or raw: true flag. Something like this should work:
let data = await sequelize.query(sql, {type: sequelize.QueryTypes.SELECT});
When I tried this, "data" was an array of two objects, each being the query result. So, the properties can be accessed by using index [0].... e.g.
data[0].forEach((item) => {
console.log(item['id'], item.id); // <-- output says "undefined, undefined"
});
Not yet sure WHY this occurs!
EDIT - it's because .query() should have only two arguments. Changing the call to: sequelize.query(sql, {raw: true, type: sequelize.QueryTypes.SELECT}) resulted in data being a single array (as expected).
Finally I was able to find the solution for it.
You just need to make a new array and push data into it by finding bases on key name like this:
suppose we have data in students object:
let finalArray = new Array();
for (var k in students ) {
finalArray.push(students[k])
}
console.log(finalArray) // Normal JSON array object :)
m.sequelize.query(sql, {
model,
mapToModel: true
})
.then(model => res.status(200).send(model))
.catch(error => res.status(400).send(error.toString())
})
I am trying to add a new key-value pair to the already loaded JSON Array. I am adding the new key-value pair to customize the header column cells in react bootstrap table but getting the below errors. Can any one please help?
'Columns' in the below state is where I wanted to add new key-value pair
state = {
data: MYResult.Products || [],
actualData: MYResult.Products || [],
columns: MYResult.ParametricList_Attributes || [],
isCompareClicked: false,
isDisabled: true,
selected: []
};
This is how I am adding the key-value pair -
componentDidMount(){
checkbox = (column, colIndex) => {
return (
<h5>{ column.text }<checkbox/></h5>
);
}
console.log(this.state.columns) ;
newColumn = this.state.columns.map((column) => {
return {...column, headerFormatter: checkbox};
});
this.setState({columns: newColumn });
}
Full code here - https://codesandbox.io/s/o1r988qkz Please uncomment the componentDidMount() to see the issue
Firstly, there's a typo in dcolumn and column.
And regarding the not defined error, you need to define it using const. Use like:
const checkbox = (column, colIndex) => {
return (
<h5>{column.text}<checkbox /></h5>
);
}
JavaScript variables need to be declared when they are used. Public class syntax can not be used everywhere. The error you're getting is self-evident - 'checkbox is not defined'.
Refer this on how to use it: https://tylermcginnis.com/javascript-private-and-public-class-fields/
I simply declared the undeclared variables in your example and the code worked.
I'm scraping a shoutbox which is limited to 10 messages; it's asynchronous and when the 11th item appears the first one is gone.
I set up a puppeteer, it scrapes the structure correctly as an array, which I dump to mongodb. The easiest way automating this I came up with is running script with the watch command and static interval.
The question is how to skip duplicates items in log, items shouldn't be unique, just don't dump the same twice. And there's probably a better way to cycle this process.attached screenshot
You can use db.collection.distinct() in MongoDB to obtain the distinct messages from your database:
db.messages.distinct( 'message' );
Alternatively, you can use db.collection.createIndex() to create a unique index in your database so that the collection will not accept insertion or update of a document where the index key value matches an existing value in the index:
db.messages.createIndex( { 'message' : 1 }, { 'unique' : true } );
In your Puppeteer script, you can use page.evaluate() in conjunction with the Set object to obtain distinct messages from the web page that you are scraping:
const distinct_messages = await page.evaluate( () => new Set( Array.from( document.querySelectorAll( '.message' ), e => e.textContent ) ) );