Query huge amount of data

Forum|Forum|3 years ago
August 23, 2022
7 replies
1625 views

Roy
Participating Frequently

Hello,

I am building a dashboard app. I came across an issue with the monday API limits that I wonder what is the best approach to solve it:
In one of the features (filtering option), I need to get all possible names (value) in a specific text column.
The issue is that the board contains around of 4500~ items. From my understanding, I need to reach the items first in order to get the names in the text column. The issue is the API not allowing me to reach so much data at once because there are so many items.

What should I do in order to get all the names in the text column without the api crashing?

Thanks!

This topic has been closed for replies.

Matias.Monday
monday.com Team Member
Forum|Forum|3 years ago
August 23, 2022

Hello there @Roy!

It looks like in this case, a good solution for this could be using pagination.

Here is an article explaining the different things you can do in order to avoid hitting the rate limit.

Hope it helps!

Cheers,
Matias

Like

R

Roy
Author
Participating Frequently
Forum|Forum|3 years ago
August 23, 2022

Hello Matias and thank you for the quick reply.

Pagination sounds great solution but I wonder how can I implement it via the API.
Eventually, I need to access all the column values in the 4500~ items on the board - so, is it possible to create a request to query items between 100 to 200 for example?

Like

Matias.Monday
monday.com Team Member
Forum|Forum|3 years ago
August 24, 2022

Hello @Roy!

You can use a query like this one:

{
  boards(ids: 1122334455) {
    items(limit: 100, page: 1) {
      column_values {
        value
        text
      }
    }
  }
}

Then you would go to page 2, then page 3 and so on until the response you get has less than 100 items, and then you will know there are no more pages to go through.

If your board is very big (lots of columns, lots of mirror columns) you might want to play with the limit. The smaller the limit, the smaller is the chance of hitting a timeout (which is probably the issue here).

Cheers,
Matias