I use laravel 5.6
I have a json file containing 500 thousand records. I want to create a logic to check whether the id of each record already exists or not in the database. If it doesn't already exist, then there will be a data insert process. If it already exists, there will be a data update process. Before updating the data, it will check whether last_modified_date in json file and database is the same or different. If it's different then it will update
I have made logic. I just want to make sure whether my logic is effective or not
My logic code like this :
$path = storage_path('data.json');
$json = json_decode(file_get_contents($path), true);
foreach ($json['value'] as $value) {
$last_modified_date = \Carbon\Carbon::parse($value['Last_Modified_Date']);
$data = \DB::table('details')->where('id', '=', $value['Code'])->get();
if ($data->isEmpty()) {
\DB::table('details')->insert(
[
'id' => $value['Code'],
'number' => $value['Number'],
'last_modified_at' => $last_modified_date,
...
]
);
}
else {
\DB::table('details')
->where('id', '=', $value['Code'])
->where('last_modified_at', '<>', $last_modified_date)
->update([
'id' => $value['Code'],
'number' => $value['Number'],
'last_modified_at' => $last_modified_date,
...
]);
}
}
The code is working. But the process seems really long
Do you have another solution that is better?
Update
I find another solution use updateOrCreate
I try like this :
$path = storage_path('data.json');
$json = json_decode(file_get_contents($path), true);
foreach ($json['value'] as $value) {
Details::updateOrCreate(
[ 'id' => $value['Code'] ],
[ 'number' => $value['Number'], 'last_modified_at' => $last_modified_date, ... ]
);
}
What do you think?
you can not use <> in updateOrCreate
i hope this code could help you:
$path = storage_path('data.json');
$json = json_decode(file_get_contents($path), true);
foreach ($json['value'] as $value) {
$detail = Details::firstOrCreate(
[ 'id' => $value['Code'] ],
[ 'number' => $value['Number'], 'last_modified_at' => $last_modified_date, ... ]
);
if($detail->last_modified_at != $last_modified_date) {
$detail->update([
'number' => $value['Number'],
'last_modified_at' => $last_modified_date,
...
]);
}
}
Related
i have this json data come from a url:
[
{
"id": "4070",
"title": "Speedway Racing",
"description": "Drivers start your engines! Professional oval track racing that explodes with stock car racing action!",
"instructions": "Use WASD or Arrow keys to move the car ",
"url": "https://html5.gamemonetize.com/b7dzsckxzmy5z3to54nif435exdiogkz/",
"tags": "Car, Cars, Racing",
"thumb": "https://img.gamemonetize.com/b7dzsckxzmy5z3to54nif435exdiogkz/512x384.jpg",
"width": "960",
"height": "600",
"category": "Driving"
},]
I need to save all data in database but for category i need to create new category from the category fled and save it in post_categry table (cat_id, post_id) if his not exists, also check f this game exists in db if yes ignore if no save it.
My GameController:
the json data come from this lik:
This link
public function importer_start(Request $request)
{
$game_cat = new Game;
$validatedata = $this->validate($request, [
'game_source' => 'required',
'source_url' => 'required|url',
]);
$dataurl = $request->input('source_url');
$sourve_name = $request->input('game_source');
$content = file_get_contents($dataurl);
$data = json_decode($content, true);
$query_ammount = $request->input('source_url');
parse_str($query_ammount, $params);
$game_amount = $params['amount'];
$query_ammount = $request->input('source_url');
parse_str($query_ammount, $params);
$game_amount = $params['amount'];
$url = $request->input('source_url');
$parmt_url = parse_url($url);
parse_str($parmt_url['query'], $params);
if ($params['format'] != 'json') {
toastr()->error('Please select JSON file to import games');
return redirect()->back();
}
if (#strpos($dataurl, Constants::GAMESOURCE_GAMEMONETIZE)) {
$finalarray = array();
foreach ($data as $key=>$value) {
// dd($value);
array_push($finalarray,array(
'game_id' => $value['id'],
'title' => $value['title'],
'description' => $value['description'],
'instructions' => $value['instructions'],
'url' => $value['url'],
'category' => !empty($value['category']) ? $value['category']: 'uncategorized',
'tags' => $value['tags'],
'thumb' => $value['thumb'],
'width' => $value['width'],
'height' => $value['height'],
'source' => Constants::GAMESOURCE_GAMEMONETIZE,
'views' => 15,
)
);
}
$categories[] = $value['category'];
Game::insert($finalarray);
toastr()->success('You have add '.$game_amount.' from '.\Constants::GAMESOURCE_GAMEMONETIZE.'');
return redirect()->route('admin.importer');
}elseif (#strpos($dataurl, Constants::GAMESOURCE_GAMEDISTRIBUTION)) {
$result = json_decode(json_encode($data), true);
$key = array_search('categoryList', array_column($result, 'categoryList'));
$finalarray = array();
foreach ($data as $key=>$value) {
array_push($finalarray,array(
'title' => $value['title'],
'gameType' => $value['gameType'],
'gameMobile' => $value['gameMobile'],
'mobileMode' => $value['mobileMode'],
'instructions' => $value['instructions'],
'description' => $value['description'],
'width' => $value['width'],
'height' => $value['height'],
'url' => $value['url'],
'source' => Constants::GAMESOURCE_GAMEDISTRIBUTION,
'assetList' => json_encode($value['assetList']),
'tagList' => json_encode($value['tagList']),
'category' => $value['categoryList'][0]['name'],
'views'=> 15,
)
);
}
//dd($finalarray);
Game::insert($finalarray);
toastr()->success('You have add '.$game_amount.' from '.\Constants::GAMESOURCE_GAMEDISTRIBUTION.'');
return redirect()->route('admin.importer');
}
}
Laravel version: 7.0
Thank you for your helpe.
I use Select2 widget in multiply mode in YII2 framework. "kartik-v/yii2-widget-select2": "#dev" - this one I have downloaded via composer.
kartik-v/yii2-widget-select2 dev-master dd09e46
I added initial values with ajax on widget init ('initSelection'). And added another ajax method to suggest new values on user's typing. When user select one from list, it replaced initial values what were added on init. New values replace initial values, but don't another new.
I want new values to add to initial instead of replace it.
<?= $form->field($model, 'security[]')->widget(Select2::class, [
'attribute' => 'security',
'hideSearch' => true,
'data'=>$security_data,
'options' => [
'placeholder' => 'Security',
'multiple' => true,
],
'pluginOptions' => [
'allowClear' => true,
'minimumInputLength' => 1,
'ajax' => [
'url' => Url::toRoute([ '/admin/security/select-items' ]),
'dataType' => 'json',
'data' => new JsExpression('function(params) { return {q:params.term}; }'),
'results' => new JsExpression('function(data,page) { return {results:data.results}; }'),
],
'initSelection' => new JsExpression('function(element, callback) { var id = '.Yii::$app->request->getQueryParams()['id'].';if(id !== "") {$.ajax("' . \yii\helpers\Url::toRoute([ '/admin/security/init-items' ]) . '", {data: {id: id},dataType: "json"}).done(function(data) {callback(data.results);});}}'),
],
]); ?>
And here is my api methods:
public function actionSelectItems($q = null){
Yii::$app->response->format = Response::FORMAT_JSON;
$out = ['results' => []];
if(!empty($q)){
$items = Security::find()->where(['like', 'title', $q])->all();
foreach ($items as $item){
$out['results'][] = ['id'=>$item->id, 'text'=>$item->title];
}
}
return $out;
}
public function actionInitItems($id = null){
Yii::$app->response->format = Response::FORMAT_JSON;
$adv = Adv::findOne($id);
$security = #json_decode($adv->security, true);
$out = ['results' => []];
foreach ($security as $item){
$text = Security::findOne($item)->title;
$out['results'][] = ['id'=>$item, 'text'=>$text];
}
return $out;
}
Is there some sort of settings or I missed something when handle http result?
I am using wbraganca dynamic form widget. It works fine for the Create action.
Let me thanks for those guys making great tutorial video on youtube!!!
I am working on the Update action now. I work it on a purchase order function.
the controller of update action :
public function actionUpdate($id)
{
$model = $this->findModel($id);
$modelsItem = $model->purchaseitems;
if ($model->load(Yii::$app->request->post()) ) {
$oldIDs = ArrayHelper::map($modelsItem, 'purchaseitem_id', 'purchaseitem_id');
$modelsItem = Model::createMultiple(Purchaseitem::classname(), $modelsItem);
Model::loadMultiple($modelsItem, Yii::$app->request->post());
$deletedIDs = array_diff($oldIDs, array_filter(ArrayHelper::map($modelsItem, 'purchaseitem_id', 'purchaseitem_id')));
$valid = $model->validate();
$valid = Model::validateMultiple($modelsItem) && $valid;
if ($valid) {
$transaction = \Yii::$app->db->beginTransaction();
try {
if ($flag = $model->save(false)) {
if (! empty($deletedIDs)) {
Purchaseitem::deleteAll(['purchaseitem_id' => $deletedIDs]);
}
foreach ($modelsItem as $modelItem) {
$modelItem->purchaseitem_purchaseorder_id = $model->purchaseorder_id;
$modelItem->purchaseitem_description = Inventory::findOne($modelItem->purchaseitem_inventory_id)->inventory_partno;
if (! ($flag = $modelItem->save(false))) {
$transaction->rollBack();
break;
}
}
}
if ($flag) {
$transaction->commit();
return $this->redirect(['view', 'id' => $model->purchaseorder_id]);
}
} catch (Exception $e) {
$transaction->rollBack();
}
}
//return $this->redirect(['view', 'id' => $model->purchaseorder_id]);
} else {
return $this->render('update', [
'model' => $model,
'modelsItem' => (empty($modelsItem)) ? [new Purchaseitem] : $modelsItem
]);
}
}
But I think the problem may happen on the view file, as the Select2 field can show the value, which is the 'id' of the product rather than the product code.
view:
<div class="panel panel-default">
<div class="panel-body">
<?php DynamicFormWidget::begin([
'widgetContainer' => 'dynamicform_wrapper',
'widgetBody' => '.container-items',
'widgetItem' => '.item',
'limit' => 50,
'min' => 1,
'insertButton' => '.add-item',
'deleteButton' => '.remove-item',
'model' => $modelsItem[0],
'formId' => 'dynamic-form',
'formFields' => [
'purchaseitem_inventory_id',
'purchaseitem_qty',
'purchaseitem_cost_usd',
'purchaseitem_deliverydate',
],
]); ?>
<?php foreach ($modelsItem as $i => $modelItem): ?>
<div class="item">
<?php
// necessary for update action.
if (! $modelItem->isNewRecord) {
echo Html::activeHiddenInput($modelItem, "[{$i}]purchaseitem_id");
}
?>
<div class="row">
<?= $form->field($modelItem, "[{$i}]purchaseitem_inventory_id")->widget(
Select2::classname(), [
'pluginOptions' => [
'allowClear' => true,
'minimumInputLength' => 2,
'language' => [
'errorLoading' => new JsExpression("function () { return 'Error on finding results...'; }"),
],
'ajax' => [
'url' => Url::toRoute('inventory/inventorylist'),
'dataType' => 'json',
'data' => new JsExpression('function(params) { return {q:params.term}; }')
],
'escapeMarkup' => new JsExpression('function (markup) { return markup; }'),
'templateResult' => new JsExpression('function(purchaseitem_inventory_id) { return purchaseitem_inventory_id.text; }'),
'templateSelection' => new JsExpression('function (purchaseitem_inventory_id) { return purchaseitem_inventory_id.text; }'),
],
])->label(false) ?>
<?= $form->field($modelItem, "[{$i}]purchaseitem_qty")->textInput(['maxlength' => true])->label(false) ?>
<?= $form->field($modelItem, "[{$i}]purchaseitem_cost_usd")->textInput(['maxlength' => true])->label(false) ?>
<?= $form->field($modelItem, "[{$i}]purchaseitem_deliverydate")->widget(
DatePicker::className(), [
'options' => [
['placeholder' => 'Please enter delivery date'],
],
'removeButton' => false,
'pluginOptions' => [
'autoclose'=>true,
'format' => 'yyyy-mm-dd',
'todayHighlight' => true,
]
]
)->label(false) ?>
</div>
</div>
<?php endforeach; ?>
</div>
</div>
I have a thought that the problem maybe related to that few lines of JsExpression function.
'escapeMarkup' => new JsExpression('function (markup) { return markup; }'),
'templateResult' => new JsExpression('function(purchaseitem_inventory_id) { return purchaseitem_inventory_id.text; }'),
'templateSelection' => new JsExpression('function (purchaseitem_inventory_id) { return purchaseitem_inventory_id.text; }'),
For the Select2 query URL method is here:
public function actionInventorylist($q = null, $id = null) {
Yii::$app->response->format = yii\web\Response::FORMAT_JSON;
$out = ['results' => ['id' => '', 'text' => '']];
if (!is_null($q)) {
$query = new Query;
$query->select('inventory_id AS id, inventory_partno AS text')
->from('inventory')
->where(['like', 'inventory_partno', $q])
->limit(10);
$command = $query->createCommand();
$data = $command->queryAll();
$out['results'] = array_values($data);
}
elseif ($id > 0) {
$out['results'] = ['id' => $id, 'text' => Inventory::find($id)->inventory_partno];
}
return $out;
}
I can load the record, when I click in the update view. Most of the data are feed in right place of the form, except the 'partno' field. I use Select2 to let user select partno by text and store the 'id' in table. It works on the Create view.
but in the update view, it only show the 'id' instead of the 'partno'.
if I make input to the field, I can select 'other' partno only, let me explain here:
if there are 2 code, "ABC" with 'id' is 1, "XYZ" with 'id' 2.
the record original is "ABC", the field show "1".
If I input "XYZ", it will show "XYZ" as normal effect of widget. But, if I change back to "ABC", it will show "1" instead of "ABC".
And the form also cannot submit for update. the button click with no effect.
I am new to Yii2 framework, and quite stuck on this issue, does anyone knows how can I solve this?
THANKS!!!!
I just solve the problem, a few issue happened actually.
To solve the Select2 widget cannot display the partno instead of the ID, I find the partno by the ID and feed it with initValueText in Select2. For my case:
$partno = empty($modelItem->purchaseitem_inventory_id) ? '':Inventory::findOne($modelItem->purchaseitem_inventory_id)->inventory_partno;
$form->field($modelItem, "[{$i}]purchaseitem_inventory_id")->widget(Select2::classname(), ['initValueText' => $partno, ...
About the update POST fail issue, I got error of Getting unknown property: backend\models\Purchaseitem::id, and I found it happened on the Dynamic-form widgets Model.php line 25. That lines is $keys = array_keys(ArrayHelper::map($multipleModels, 'id', 'id'));
If I change the 'id' to my ID field name, i.e 'purchaseitem_id', it will work, but this model will only work for this Id field name afterward. So I get the model's primaryKey and make it work for my other model.
add this line $primaryKey = $model->tableSchema->primaryKey; and modify above line $keys = array_keys(ArrayHelper::map($multipleModels, $primaryKey, $primaryKey));
Tearing my hair out at this point, hopefully someone can help me out!
I am using the Kartik-V Typeahead Advanced widget with Yii2.
The plugin works, in that the functionality is working perfectly on the page, I search and the results appear in the auto complete list.
Unfortunately, I am unable to store the result in my database. I am seeing an issue on the following line:
->where([ 'name' => $model->name ])//This variable is returning null
Am I trying to store the data incorrectly? I have tried everything I can think of, but I am sure someone here will come up with something better!
See below for the full code.
My controller:
public function actionIndex()
{
$model = new Member();
if ($model->load(Yii::$app->request->post())) {
$test = Test::find()
->where([ 'name' => $model->name ])//This variable is returning null
->one();
$test->updateCounters(['times_used' => 1]);
}
return $this->render('index', [
'model' => $model,
]);
}
/*************
* Initial prefetch of results
*************/
public function actionPrefetchlist() {
$query = new Query;
$query->select('name')
->from('test_table')
->limit(10)
->orderBy('times_used');
$command = $query->createCommand();
$data = $command->queryAll();
$out = [];
foreach ($data as $d) {
$out[] = ['value' => $d['name']];
}
echo Json::encode($out);
}
/*************
* Remote results
*************/
public function actionRemotelist() {
$query = new Query;
$query->select('name')
->from('test_table')
->where('name LIKE "%' . $q .'%"')
->limit(10)
->orderBy('times_used');
$command = $query->createCommand();
$data = $command->queryAll();
$out = [];
foreach ($data as $d) {
$out[] = ['value' => $d['name']];
}
echo Json::encode($out);
}
The view file:
echo $form->field($model, 'name')->label(false)->widget(Typeahead::classname(), [
'name' => 'name',
'options' => ['placeholder' => 'Filter as you type ...'],
'pluginOptions' => ['highlight'=>true],
'dataset' => [
[
'datumTokenizer' => "Bloodhound.tokenizers.obj.whitespace('value')",
'display' => 'value',
'prefetch' => Url::to(['prefetchlist']),
'remote' => [
'url' => Url::to(['remotelist']) . '?q=%QUERY',
'wildcard' => '%QUERY'
]
]
]
]);
you ask here for a new model:
$model = new Member();
so you get a empty model
so the $model->name is empty
if you set the model $model->name='test';
than it will be filled so, fill the model first
So it turns out it was a massive rookie error.
If anyone else stumbles upon something similar, I removed the attribute name from the model's "rules()"
I need an integer in the database but I wanted users to enter in a string (I would then convert it in the controller). Removing it from the rule broke everything.
Hopefully this helps someone else :)
Before I had this:
//ArticlesController::index
$articles = $this->Articles->find('all', [
'contain' => ['Comments']
]);
So I set the fields key:
//ArticlesController::index
$articles = $this->Articles->find('all', [
'fields' => ['title', 'text],
'contain' => ['Comments']
]);
Since I set the fields key the result of the find is not bringing the comments anymore.
$articles = $this->Articles->find('all')
->select(['fields_you_want_from_Articles'])
->contain(['Comments' => function($q) {
return $q
->select(['fields_you_want_from_Comments']);
}]);