Timeout and data return issues when requesting large queries

I have noticed similar issues receiving a 504 Timeout Error when trying to retrieve large queries. You bring up very good points about issues and possible opportunities.

I would like to share something that I discovered about query requests that I have not been able to find in the documentation either:
When passing the limit parameter to a query response, subsequent queries to the same board (e.g. within a looping structure) will select the next contiguous block of items from the board, instead of starting from the beginning of the board. I haven’t been able to confirm this, but it seems that Monday.com maintains an internal cursor when a query returns a limited number of items.

When working with large boards (either a large number of items, or a large number of columns), I have found that the ability to “break up” the data that comes from the query solves the problem of the request timeout, and then I can concatenate all results locally.

@arnold_wikey yes! I’ve found the same. The two seem to work together so long as they are at the root level of your request. I’m currently pulling full pulse sets across all boards the way you are describing above. I had some weird responses having page requests anywhere except at root during testing that was a bit annoying. For example, I couldn’t successfully pull board --> item with page on item, and of course page on board just breaks up the boards. I think the pagination documentation could use some more description regarding when it does and does not work, as well as documentation regarding time outs and Monday’s recommendation for avoidance.

Hey guys,

there are API rate limits on the complexity levels which are 10,000,000 / minute per account on monday.com.
It’s higly recommended to lower the complexity level of your querys.

You can read more about rate limits here:

https://monday.com/developers/v2#rate-limits-section

@TMNXT-Dev I appreciate your response, however I think we are discussing different topics.

1. I record the cost of every query I run.
2. From what I see query costs do not align with machine workload
3. This is the most expensive query I run. It costs 2,013,010 and runs in less than a second. It costs so much because I pulled a lazy move and didn’t feel like cycling through board page, and Monday’s cost calculator multiplies by 1,000. Regardless, here it is.

4. The paginated item pulls I am running require much less cost, and still take between 15-45 seconds to run.

5. Any limit over 2000 on these item requests pushes response time too closely to the 60 second mark for comfort. When I adjust my limit to 4000 or 5000 I bump into response times that are 60+ seconds and run into timeout issues, with a minimal query cost per run time. Here’s an example, using a page number higher than what currently exists in my dataset. Minimal cost, but if the data existed the call is likely to timeout.

{complexity{query, after}, items (page: 50, limit: 5000) {board {id}id,name,state,group {id},column_values {id,text,value}}}
returns: “query”: 190010

6. When a timeout is achieved, the response is an html page body, instead of a json body with an error code. This add unnecessary complication to error handling.

Hey @morhriveion ,

I don’t represent monday, I’m a dev just like you who wants to help 🙂 .

And I’m sorry, I obviously missunderstood your question.

@TMNXT-Dev apologies, I’ll edit my previous response accordingly.

Hi @morhriveion!

Scott here from monday.com 👋

Thank you so much for taking the time to bring this to our attention. The errors being returned in HTML as opposed to JSON is something that we’re aware of and are working on resolving. I’ve passed this feedback along to the team to let them know it’s a pain point that you’re currently running into.

You’d also noted:

I’m disappointed further that pagination doesn’t interrupt query timeouts. Return an error at data transmission if anticipated query call will result in greater than 60 seconds, or just complete the query and send the data.

I just wanted to let you know that I’ve passed that feedback along to the team as well 🙂

If you have any other suggestions, please let me know! 🙌

-Scott

@morhriveion @arnold_wikey how have you guys been calculating the total pages? I’m running into a similar issue of timeout and was hoping I could loop through the items locally. Any tips or tricks? Thanks guys!

@morneluckybeard my use may be different from yours depending on the amount of columns you are calling at a time.

I used a variable to control how many rows/pulses I called at a time and dropped that down until the highest call wast no more than 20-30 seconds. I left that variable at 2000.

I don’t remember why I chose to not check for a response equal to the variable. Instead I opted to call the next page so long as the previous call returned pulses. I didn’t leave any notes on it but I’m sure either would work.

The time out issue was my biggest frustration while working with this. Definitely make sure you are not at risk of bumping into this timeout. During my first test phase I was around the 4000-5000 mark, and had zero issues. Unfortunately I do a lot of work during non-peak hours, and once I moved into any where near peak hours I ran into timeouts that broke every request sequence.

Thanks for taking the time to respond, I was hoping I could kick off all of my calls asynchronously instead of calling them sequentially, but I think I might have to do that, reduce the limit to like 200 and keep calling until I get an empty array back.

Thanks again, I really appreciate your response.