I’m working on one project where I need to transfer bunch of files to Autodesk Construction Cloud (ACC) using Forge.
But I’m curious what is limitation of max number files we can upload to particular Project.
I’ve referred below mentioned document for Product tools & limitation
https://knowledge.autodesk.com/support/docs/learn-explore/caas/CloudHelp/cloudhelp/ENU/Docs-About-ACC/files/Product-Limitations-html.html
The max files per Folder is 10,000
But There is nothing mentioned about Max files per Project.
Or does it like we can make Number of folders under One Project and each folder has 10,000 files limit?
Please help me with this..
Currently, there is no limit to the maximum number of files per project, but has limiations you found in https://knowledge.autodesk.com/support/docs/learn-explore/caas/CloudHelp/cloudhelp/ENU/Docs-About-ACC/files/Product-Limitations-html.html
Maximum files per folder: 10,000
Maximum number of subfolder levels: 25
Maximum file size: 5TB
Maximum name length for folders: 255 bytes
Maximum pages in a PDF file: 2,000
Maximum number of custom attributes per project: 1,000
Maximum rows in a files log: 20,000
Files which can't be downloaded: Data Exchanges
Related
visit resource limits - https://cloud.google.com/functions/quotas
Number of functions : The total number of functions that can be deployed per project- 1000 limit.
its means 1000 separated functions can deployed or each function deployed 1000 versions?
There can be no more than 1,000 active exported Cloud Functions per project.
The number of deployments is not relevant to this limit. But if you ever have more than 1,000 active, exported functions in a single deployment, that deployment will fail.
Whichever that comes first, same thing....Per project the limit is 1,000 times or versions total in one project.
Then each function size must be <= 100MB for sources and <= 500MB for sources plus modules.
Check here for more.
We're building a system that migrates documents from a different data store into Drive. We'll be doing this for different clients on a regular basis. Therefore, we're interested in performance, because it impacts our customer's experience as well as our time to market in that we need to do testing, and waiting for files to load prolongs each testing cycle.
We have 3 areas of drive interaction
Create folders (there are many, potentially 30,000+)
Upload files (similar in magnitude to the number of folders)
Recursively delete a file structure
In both cases 1 and 2, we run into "User rate limit exceeded" errors with just 2 and 3 threads, respectively. We have an exponential backup policy as suggested that starts at 1 second, and retries 8 times. We're setting the quotaUser on all requests to a random uuid in an attempt to indicate to the server that we don't require user specific rate limiting - but this seems to have had not impact as compared to when we didn't set the quotaUser.
Number 3 currently uses batch queries. 1 and 2 currently use "normal" requests.
I'm looking for guidance on how best to improve the performance of this system.
Our employees are doing work for clients, and the clients send us files which contain information that we turn into performance metrics (we do not have direct access to this information - it needs to be sent from the clients). These files are normally .csv or .xlsx so typically I read them with pandas and output a much cleaner, smaller file.
1) Some files contain call drivers or other categorical information which repeats constantly (for example, Issue Driver 1 with like 20 possibilities and Issue Driver 2 with 100 possibilities) - these files are about 100+ million records per year so they become pretty large if I consolidate them. Is it better to create a dictionary and map each driver out to an integer? I read a bit about the category dtype in pandas - does this make output file sizes smaller too or just in-memory?
2) I store the output as .csv which means that I lose the dtypes if I ever read the file again. How do I maintain dtypes and should I save the files to sqlite instead perhaps instead of massive .csv files? My issue now is that I literally create codes to break the files up into separate .csvs per month and then maintain a massive file which I use for analysis (dump it into Tableau normally). If I need to to make changes to the monthly files I have to re-write them all which is slow on my laptops non-SSD hard drive.
3) I normally only need to share data with one or two people. And most analysis requests are adhoc but involve like one - three years worth of very granular data (individual surveys or interactions each represented by a single row in separate files). In other words I do not need a system with high concurrency of read-write. Just want something fast, efficient, and consolidated.
Had a proposed solution from a co-worker to setup a caching layer for files that never (almost never) get updated. Currently the Ruby application has to fetch content from DB, render a page and serve for each request, with exception of caching images/css/js in Akamai CDN.
The performance is terrible when there are 5K users on the site.
Proposed solution was to generate 15 million static pages once (4T), store in a single directory on a NFS server, then share among 9 Apache/Phusion-Passenger servers, and set Apache to serve static content from mounted NFS share.
In addition to initial 15 million files, 8K static files will be generated per day and added into NFS
While, I don't believe that this is a good solution and don't feel comfortable implementing this and looking into Varnish to cache mostly accessed articles, I'd like to know what others think about the proposed solution vs varnish.
Questions:
Can 15 million files (4T) be stored in a single directory in Linux (CentOS)
Can such a large directory be shared via NFS? Will that be stable
Can 15 million files be stored in hashed directories ? or still bad idea?
Is there a max file limit for NFS share?
I'd like to thank you for your advise in advance.
You can give GlusterFS a try.
First partition your articles by category. Then store them to a GlusterFS directory like:
/mnt/articles/category1/201304/20130424/{a lot of files}
I have a 6 nodes GlusteFS cluster to store log files. Currently it has 8T+ files and increase 30G+ everyday without any problem.
We have about 60 million webpages in a compressed format. We would like to de-compress and work with these files individually.
Here are my questions!
First, if I decompress them into the file system, would the FS cope with such number of files. My file system is ext4. (I've 4 different file systems so I can divide the data between them like 15 M pages for each file system)
Secondly, Would storing these files into a relational database be a better option? assuming that all the hassle of cleaning html text is done before inserting them into the database.
Thanks,
If you extract them into a single directory you may exceed the maximum allocated indices in that folder. If you extract them into multiple directories you will fair better.
60 Million is definitely a fair amount, if you plan on doing any indexing on them or searching then a database would be your best option, you can do indexing on files using something like lucene it all depends on what you want to do with the files After they ave been extracted.
I currently have a similar issue with images on a large user site, the way I got around this issue was to give each image a GUID and for each byte in the guid assign it to a different directory, then the next byte under a subdirectory (down to 8 bytes) if my fill ratio goes up I'll create more subdirectories to compensate, it also means I can spread it across different net storage boxes.