Extract required data from huge DB about 40 GB: Taking too much time to filter data and get required results

Recommend this page to a friend!
  All requests RSS feed  >  Extract required data from huge DB ab...  >  Request new recommendation  >  A request is featured when there is no good recommended package on the site when it is posted. Featured requests  >  No recommendations No recommendations  

Extract required data from huge DB about 40 GB

Edit

Picture of Siva Kumar by Siva Kumar - 6 years ago (2015-06-19)

Taking too much time to filter data and get required results

This request is clear and relevant.
This request is not clear or is not relevant.

+2

I am working in system which connects to multiple channels like booking.com, hrs.com, centralR, etc. nearly about to 50 channels.

The working system takes input data from hotels and send them to these channels for online reservation.

There is page which shows the each update sent to channel received from hotels base for selection of hotel, room type, etc..

We maintain the last 3 months data which is nearly about 40 GB.

This DB is well designed and indexed.

Can any one please suggest a package which can improve retrieving from this data.

  • 1 Clarification request
  • 1. Picture of Manuel Lemos by Manuel Lemos - 6 years ago (2015-06-19) Reply

    I suspect there is not PHP class for that problem, if this is a matter of speed of queries to the database.

    I think you need to figure which database queries are taking too long.

    If you are using MySQL, then you can use the EXPLAIN command followed by the slow query and it will tell you what clauses are using indexes or not.

    Try to do that for your slowest query and reply here showing the query and the output of the EXPLAIN command, so we can advise.

    • 2. Picture of Dave Smith by Dave Smith - 6 years ago (2015-06-19) in reply to comment 1 by Manuel Lemos Comment

      I have to agree, that it is most likely a poorly designed query, assuming that the database is truly well designed and indexed.

      Another possibility is scalability issues, how much traffic is your site getting?

Ask clarification

1 Recommendation

QDataObject: Access MySQL query results as objects

This recommendation solves the problem.
This recommendation does not solve the problem.

+1

Picture of Manuel Lemos by Manuel Lemos Reputation 20815 - 6 years ago (2015-06-20) Comment

Looking again there is this MySQL database access wrapper that logs slow queries. This may not optimize your application automatically but it may help detecting which queries are taking too long and probably need to have the be rethought or added adequate indexes.


Recommend package
: 
: 
For more information send a message to info at phpclasses dot org.