When using Forge Data Management API endpoint
projects/:project_id/folders/:folder_id/search we have two problems.
It seems that we sometimes have to wait for several minutes (hours?)
after a model is uploaded until it can be found by the search.
We often get error 429 "Too Many Requests" even that we only call do
very few calls (less than 10 within an hour).
These issues makes the endpoint hard to use in production code. Is there anything we can do to improve the success rate? Is Autodesk going to improve the endpoint?
This question is related to How to find cloud Item id of a Revit model?
We are aware of those issues and working on improving things in those areas. However, I cannot tell when exactly those will become available.
In the meantime, depending on your workflow, there are two things that could be of help:
Use webhooks in order to be notified about new files being added to BIM 360/ACC
Use folder/contents endpoint to find the file you need, which also supports filtering just like the folder/search endpoint. You would have to iterate through subfolders though if you wanted to look for items in them as well. Newly added files should show up here straight away.
Related
I am creating a webpage in ReactJS for post feed (with texts, images, videos) just like Reddit with infinite scrolling. I have created a single post component which will be provided with the required data. I am fetching the multiple posts from MySQL with axios. Also, I have implemented redux store in my project.
I have also added post voting. Currently, I am storing all the posts from db in redux store. If user upvotes or downvotes, that change will be in redux store as well as in database, and web-page is re-rendering the element at ease.
Is it feasible to use redux-store for this, as the data will be increased soon, maybe in millions and more ?
I previously used useState hook to store all the data. But with that I had issue of dynamic re-rendering, as I had to set state every time user votes.
If anyone has any efficient way, please help out.
Seems that this question goes far beyond just one topic. Let's break it down to the main pieces:
Client state. You say that you are currently using redux to store posts and update the number of upvotes as it changes. The thing is that this state is not actually a state in your case(or at least most of it). This is a common misconception to treat whatever data that is coming from API a state. In most cases it's not a state, it's a cache. And you need a tool that makes work with cache easier. I would suggest trying something like react-query or swr. This way you will avoid a lot of boilerplate code and hand off server data cache management to a library.
Infinite scrolling. There are a few things to consider here. First, you need to figure out how you are going to detect when to preload more posts. You can do it by using the IntersectionObserver. Or you can use some fance library from NPM that does it for you. Second, if you aim for millions of records, you need to think about virtualization. In a nutshell, it removes elements that are outside of the viewport from the DOM so browsers don't eat up all memory and die after some time of doomscrolling(that would be a nice feature tho). This article would be a good starting point: https://levelup.gitconnected.com/how-to-render-your-lists-faster-with-react-virtualization-5e327588c910.
Data source. You say that you are storing all posts in database but don't mention any API layer. If you are shooting for millions and this is not a project for just practicing your skills, I would suggest having an API between the client app and database. Here are some good questions where you can find out why it is not the best idea to connect to database directly from client: one, two.
The following api link for the Node JS package shows an endpoint that limits to 60 RPM: forge-api docs
Support has said that 60RPM is only for endpoints where forceget is set to true in the options, and that the implicit value for the forceget option when calling getModelviewProperties is set to false, yet I'm still getting a limited to 60 requests per minute.
Even when setting forceget to false it limits the rpm.
How can I have more RPM for this endpoint?
Unfortunately these endpoints have to be limited for reasons. But I agree 60rpm with forceget=true is a very low limit. However, forceget=false has a 14000rpm limit. Note as well that forceget=true is ignored if the resources are still available in the cache from a previous call.
That means that you first call with forceget=false, and if there is a problem, you call the resource with forceget=true only once, and give it some time for the resources to be loaded in the cache. All following calls should be using forceget=false. That way, you would save your forceget=true quota.
If you still have problems, you can either let me know at cyrille at autodesk.com with an example, and I'll debug the problem with you.
Or an alternative is to parse/expand the db files yourself, which can be a very good option if you need to improve your application performance a lot. I got an old sample doing this here which is more powerful than the Forge API for querying properties. I am currently rewriting it in typescript in a new sample, but this one isn't ready yet - it will be posted here.
I am currently in the process of implementing pagination, sort and search functionalities in the project files/plans/sheets views of BIM 360 Docs integration.
Since I couldn't find any best practices regarding to these features, I thought I would reach out so that I don't keep stuck reinventing the wheel.
Background:
Most of the implementation uses https://github.com/Autodesk-Forge/forge-api-dotnet-client/ SDK.
Based on what I saw, pagination in Autodesk API is very basic and does not play well with filtered views. Please correct me if I am wrong, but it looks like there is no way to get number of items in the view and/or calculate total number of pages in the resultset.
If one uses filtering to limit types of items returned by the API (e.g. documents, sheets, project files), API applies pagination first and filters second. That causes holes in returned resultsets, e.g. one would request page 1 sized as 5 items, and get 3 items back, then request similarly sized page 2 and get no items back, then page 3 would yield 2 items.
The above mentioned issues force us to use dynamic lazy-loading paging, similar to how it's currently done in the BIM360 Docs UI.
Question:
Is there a different, better way to paginate? Or do we have to lazy-load results while scrolling, never knowing how much records the next page would return?
Unfortunately, paginating is not available for Forge MD API of BIM360 currently as I know. Apologizing for any inconvenience caused.
However, it's been logged as request id FDM-1769 a few days ago. I saw your name on the request list. So I think it will be supported in the future. Besides, a workaround is to fetch all data from the API, then paginate on the client side via Javascript.
The problem I'm posed with is backfilling a specialized database, using data from the event log of a given smartcontract on an Ethereum blockchain.
The question is however: how to do so without reaching the limits of eth_getLogs (also without limits: how to have reasonably sized RPC responses)
What I tried so far
I prefer to use Infura, but they limit this call at 100 entries per response. And rightfully so, querying should be done in small chunks for load balancing etc. Is api pagination + eth_getLogs the right way to collect data for backfills?
Idea 1: eth_getLogs on ranges of blocks
I don't know of any way to paginate the eth_getLogs other than querying for ranges of blocks. A block may contain more than 100 events however, which prevents me from reading all of the data when using Infura. Maybe there is a way to paginate on log index? (100 is something I came accross when experimenting, but I can't find documentation on this)
Idea 2: log filters
Using a filter RPC call is another option: i.e. start a "watcher" on a range of old blocks. I tried this, but the Infura websocket RPC I am using doesn't seem to give any response, and neither does Ganache when testing locally. Non-archive (i.e. live watching) logs work, so I known that my code is working as intended at least. (My go-ethereum Watch... generated binding call works, but does not result in responses on the output channel when specifying an old block in bind.WatchOpts.Start)
Does anyone have any suggestions on how to retrieve large amounts of log data? Or a link to other projects that tackled this problem?
My real-time document allows the user to edit the file name within the editor (much like Google's own apps). I represent this as a collaborative string so all collaborators see the file renames as soon as possible.
I'm trying to determined the best and most efficient way to keep this collaborative string in sync with the actual file name. There are two scenarios to consider:
In Editor Changes
If a user edits the document name within the editor. In this case we need to use the Drive API to push that change out to the file on Google drive. To avoid race conditions, it is best if only one of the collaborators pushes the change out. The easiest way to do this seems to check if the rename event was local.
I also found it best to add a delay so we are not pushing the rename out to the Drive API with every character change. If a few seconds pass with no more name changes at that point it pushes the change out. This all seems to work well.
External Changes
The harder one and the one I am interested in requesting advice on, the case when the file name is changed externally. For example, if the user renamed the file within the Drive interface itself. We want this change to update our collaborative string to match.
My application is entirely client-side so I can't use webhook push notifications. So my only solution is to poll the file name every X seconds (currently set to 10). But this presents the following problems:
It is API intensive. If you have 4 collaborators that keep the screen open for 8 hour that is 11520 API calls. If my app has lots of users with lots of documents I could see how this might push me past my API limits.
To avoid race conditions (and reduce API calls) we only want one collaborator to check for changes and update the collaborative string if the file name has changed. But how to pick when collaborators might join/exit at any time? Currently I am having each collaborator check anytime the collaborators change if they are the "leader". The "leader" is the collaborator whose session id is the highest. This seems to work but it all seems fairly hackey. Also if collaborators join close together I wonder if it might be possible that a race condition would cause multiple collaborators to think they are the leader.
Is there an easier way? An real-time API function I am missing?
It would be ideal if the real-time API just provided a method that stored the document name. Anytime the real-time API checks for mutations it could grab the latest document name.
I think you've identified the options. There isn't any built in functionality currently to sync it via the Realtime API specifically.
Personally I'd probably back off the poll time a lot.. its probably not critical that the title is always exactly up to date, so asking every few minutes is probably sufficient and would greatly reduce your qps.
In terms of identifying a "leader", I can't think of anything better than something deterministic based on the session id. So long as each rechecks on each session join/leave event, I don't think there should be any issues.