PHP Classes
PHP Classes
elePHPant
Icontem

How to Use Queue To Speedup PHP Processing Tasks Part 1: Queueing Slow Tasks

Recommend this page to a friend!
  Blog PHP Classes blog   RSS 1.0 feed RSS 2.0 feed   Blog How to Use Queue To S...   Post a comment Post a comment   See comments See comments (6)   Trackbacks (0)  

Author:

Posted on:

Categories: PHP Tutorials

Queues are an important solution, especially when need to create a processing task that may take a long time to complete and the user or the creating process cannot wait until the task is finished.

This is the case for instance of sending newsletter email messages to many users.

Read this article to learn how queues work and how you can implement one in PHP using a simple database.




Contents

The Importance of Queues to Improve Performance

Implementing Queues in PHP

Conclusion


The Importance of Queues to Improve Performance

Queues are everywhere. When you delete a photo on your iPhone, it goes to the Bin and will be deleted from there after 30 days. That is a queue. Without it you will not be able to recover a picture removed by mistake.

When you work on a plane and are able to write 3 or 4 emails offline, have them saved in outbox and get them sent when you get to the WIFI connection in airport, that is a queue. Without it you will not be able to write several messages offline, the mailing program would just complain that connection is not available after you finished your very first email and would not allow you to create the second one.

Literally, a queue is also a long line of people in McDonalds. Several managers just take a next customer and serve them their order. If there was just a single manager, the queue processing would take ages and huge profit loss.

So a queue allows the work to be shared among multiple "processors" (also known as "agents" or "workers"). So, these 3 examples just illustrate the basic principles of queues:

  1. Queues make the user experience better

  2. Queues allow to postpone the heavy processing and thus make the interface more robust

  3. Queues allow separation of work between many workers

No surprise that if you apply queues in your application, its development will become more joyful and the application will become more robust.

A quick example is sending a notification email after a successful purchase. If you make your user wait until the email gets sent in real time and your email server is slow, your user will get not as happy as he or she could be if the interface would quickly say "Hey, thanks for purchase".

To achieve that speed of interface response, you just need to save the notification into your notifications queue and process it later soon, but not immediately. Magic!

So this article is about how to work with queues (easy!), and it consists of 2 parts with PHP examples:

  1. How to fill a queue

  2. How to process the queue.

Implementing Queues in PHP

So, let's clarify what a queue is: technically it is a data structure where data comes from one end and leaves the queue from another end. An example is a car wash and a line of cars: the first car from the line gets into the car wash and then leaves it, then the second car gets in, and so on.

That is why the queue is also called a FIFO: "first in, first out" in contrast to a stack, another data structure, one which "last in, first out" approach is used (your browser history for example).

Since queues are that simple by definition, you can use almost any kind of data storage. You can even use a usual PHP array:

<?php
class Queue {

protected $_storage = array();

public function addItem( $item ) {
$this->_storage[] = $item;
}

function getItem() {
return array_shift($this->_storage);
}

};
?>

That would work if your queue has a very short "time to live", within a single script call. The $queue variable will not exist if you refresh the page.

So you need a persistent data storage container and PHP adapter to access it. From my experience that could be:

  • A plain text file, a log file, why not?

  • Gmail inbox, say, if you need to process email messages that keep and keep coming, then you don't have to care about filling the queue though.
  • Special queues solutions like Apache ActiveMQ or Apache Kafka
  • A web-service based storage like iron.io
  • A database table, I think, one of the easiest and available solutions
Let's say your application has to send an email to a user after a purchase.

Step 1. Create a Queue

Let's see an example of a database table, it's very easy. You just need to create a table called queue, where every line is a 'job' to process, and the table has these fields:

  • id: this is an unique ID of a task

  • data: the task data to process. Can be image path to process, email to sent or a domain name to register what your business is about
  • status: a list of possible states of a job: new, processing, done, failed. Oh believe me, you will use this field a lot.
  • error_message: optional field for more info when task processing failed: out of space, credit or time will be visible here.

Here I also recommend to have some additional fields:
  • created_at: timestamp when the item was queued, it set once and never changes
  • updated_at: timestamp when the item was touched or processed, it will change at list couple of times
  • agent_id: only if you have more then one script that processes the queue (covered in 2nd part of this article)

Step 2. Implement your Queue Adapter.

<?php
class Queue {

public function addItem($item) {

$query = 'INSERT INTO (status, data, created_at) my_queue VALUES ("new", "' . json_encode( $item ) . '", NOW())';
sql_run_query($query);
}
};

?>

Step 3. Find in Your Code where the Heavy Operation is (sending the email):

<?php
//this line is the heavy operation and can take up to 10 seconds of waiting since our mail server is slow
Mail::send(array(
'to' => 'user@server.com',
'subject' => 'Thanks for your purchase',
'body' => '<b>Thank your very much, here are the details...</b>'
));
echo 'Thanks for your purchase!';
?>

Step 4. Queue the Heavy Operation

<?php
//this is now a super-fast operation since it's just a database INSERT, no real email is sent now
$QueueManager->addItem(array(
'to' => 'user@server.com',
'subject' => 'Thanks for your purchase',
'body' => '<b>Thank your very much, here are the details...</b>'
));
echo 'Thanks for your purchase!';
?>

Conclusion

So, as you can see, working with queues if very simple and can make your life easier.

In next chapter I will show how to process items and how to divide work between many workers.

If you liked this article or you have question regarding creating queue or queuing tasks, post a comment here.




You need to be a registered user or login to post a comment

1,444,733 PHP developers registered to the PHP Classes site.
Be One of Us!

Login Immediately with your account on:

FacebookGmail
HotmailStackOverflow
GitHubYahoo


Comments:

5. very cool - ololade tobi (2015-12-16 16:20)
waiting for the second part... - 0 replies
Read the whole comment and replies

4. Php queue - sulay njiaye (2015-12-02 17:14)
Php queue... - 0 replies
Read the whole comment and replies

3. Looks fine - Len (2015-12-01 18:39)
Where is step 2... - 0 replies
Read the whole comment and replies

2. Very interesting - des (2015-12-01 18:38)
Cannot wait for more... - 0 replies
Read the whole comment and replies

1. Good Work - Mohammed Lubbad (2015-11-30 18:57)
Continue please... - 1 reply
Read the whole comment and replies



  Blog PHP Classes blog   RSS 1.0 feed RSS 2.0 feed   Blog How to Use Queue To S...   Post a comment Post a comment   See comments See comments (6)   Trackbacks (0)