I'm creating a questionnaire application in Qt, where surveys are created, and users log on and complete these surveys. I am saving these as JSON.
Each survey could have 60+ questions, and are completed multiple teams by different people.
Is it more appropriate to save as 1 JSON file, or a file for each Survey?
I would use a Database rather than a JSON file. You can use JSON to serialize data and transfer it through processes and computers or servers, but you don't want to save big data to a JSON file.
Anyway if that's what you want to do I would save each survey in a different JSON file. Maybe keep them in order by assigning a unique identifier to each file (name of the file) so that you can find and search for them easily.
One single file would be a single point of failure, and when reading and writing it there would be concurrency problems. One file for each survey should soothe the problem.
Related
I am wondering of what is the best practise for handling new version of the same model in the Data Management API Bucket system
Currently, I have one bucket per user and the files with same name overwrites the existing model when doing a svf/svf2 conversion.
In order to handle model versioning in be the best manner, should I :
create one bucket per file converted
or
continue with one bucket per user.
If 1): is there a limitation of number of buckets which is possible to create?
else 2): How do I get the translation to accept an bucketKey different than the file name? (As it is now, the uploaded file need to be the filename to get the translation going.)
In advance, cheers for the assistance.
In order to translate a file, you do not have to keep the original file name, but you do need to keep the file extension (e.g. *.rvt), so that the Model Derivative service knows which translator to use. So you could just create files with different names: perhaps add a suffix like "_v1" etc or generate random names and keep track of which file is what version of what model in a database. Up to you.
There is no limit on number of buckets, but it might be an overkill to have a separate one for each file.
I am trying to copy files from one folder to another. However source folder has multiple folders in it and then multiple files. My requirement is to move all the files from each of these folder into single folder. I have about millions of file and each of these files have hardly 1 or 2 records.
Example -
source_folder - dev-bucket/data/
Inside this source_folder, I have following -
folder a - inside this folder, 10000 json files
folder b - inside this folder, 10000 json files
My aim - Target_folder - dev-bucket/final/20000 json files.
I tried writing below code, however, the processing time is also huge. Is there any other way to approach this?
try:
for obj in bucket.objects.filter(Prefix=source_folder):
old_source = {'Bucket': obj.bucket_name,'Key': obj.key}
file_count = file_count+1
new_obj = bucket.Object(final_file)
new_obj.copy(old_source)
except Exception as e:
logger.print("The process has failed to copy files from sftp location to base location", e)
exit(1)
I was thinking of merging the data into 1 single json file before moving the file. However, I am new to Python and AWS and am struggling to understand how should I read and write the data. I was trying to do below but am kind of stuck.
paginator = s3_client.get_paginator('list_objects_v2')
pages = paginator.paginate(Bucket=s3_bucket, Prefix=FOLDER)
response = []
for page in pages:
for obj in page['Contents']:
read_files = obj["Key"]
result = s3_client.get_object(Bucket=s3_bucket, Key=read_files)
text = result["Body"].read().decode()
response = response.append(text)
Can you please guide me? Many thanks in advance.
If you need copy one time, I sugget to use aws cli
aws s3 cp source destination --recursive
https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html
If possible, it is best to avoid having high numbers of objects. They are slow to list and to iterate through.
From your question is seems that they contain JSON data and you are happy to merge the contents of files. A good way to do this is:
Use an AWS Glue crawler to inspect the contents of a directory and create a virtual 'table' in the AWS Glue Catalog
Then use Amazon Athena to SELECT data from that virtual table (which reads all the files) and copy it into a new table using CREATE TABLE AS
Depending upon how you intend to use the data in future, Athena can even convert it into a different format, such as Snappy-compressed Parquet files that are very fast for querying
If you instead just wish to continue with your code for copying files, you might consider activating Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects. Your Python program could then use that inventory file as the input list of files, rather than having to call ListObjects.
However, I would highly recommend a strategy that reduces the number of objects you are storing unless there is a compelling reason to keep them all separate.
If you receive more files every day, you might even consider sending the data to an Amazon Kinesis Data Firehose, which can buffer data by size or time and store it in fewer files.
I'm currently thinking of a concept for a react-native app where people add events over a period of time like a diary/log. These events need to be exported and for security and privacy reasons I don't want to use a database. I've read you can use JSON files as a storage method, too.
How can I store data from the app to a JSON file and load the data from the JSON file back in the app? Don't need any code, helpful articles or webpages are appreciated
Assuming that you already have all the app data into a json, its rather simple.
Decide where to store the appdata.json, lets call it APP_DATA_PATH
Find a library to read/write files (I've only used expo-file-system)
On app boot, check if APP_DATA_PATH exists, if it does read the file and load into app, if not assume its a new user
Whenever app data changes, write the changes to APP_DATA_PATH
Here's some background information to help understand my dilemma. I've got a system written in PHP where users can fill out a form, upload a few images and everything gets archived in a .zip, I then have an "Account Page" that echo's a simple table containing a list of all of that specific users uploaded .zip's, the reason I'd need separate filepaths for each and every .zip is because no matter the user, they all need to be in the same single folder on the server for the broader application to work (these .zips are used for a mobile based Package Manager; see Cydia).
Would the smarter choice be to have every name of each .zip stored in a table on the database with things like the username and user_id tying to each entry, which would result in a few entries being for one user, and a few more being for a different user (depending on upload time), and everything will be unorganized. If I need to enter the database and manually search for a specific user's uploaded .zip's, I'll have to sort them out by user_id.
OR
Would it be more cost effective (in terms of passing data and hardware limitations) to have an XML file or even a plain .txt file on the server with a list of the users uploaded .zip's, and an exact file path pointing to them, and in the database simply have one column tied to the rest of the users information that points to that file containing multiple lines of filepaths?
It's a mouth full I know, and I have no problem achieving either of them as it's well within my skillset and knowledge of PHP & MySQL, I'm just not too sure which is the most secure and the best practice in the industry for this type of system.
I have JSON data which has say three levels of nesting.I need to support localization of my app.I can find resources on how to create res file and support multiple languages,But they all seem to be for a single string.How do I place my nested JSON data in res file??
There are two simple ways to solve the problem that may work for you:
Depending on the size of the JSON file, you can just store the stringified JSON data as a string resource. For small amounts of data, it's a simple solution.
For larger files, I'd suggest instead that you store a file name/path which points at the corresponding JSON file that is appropriate for the current locale. You'd just load the file based on the file name provided in the resource at run time.