Hello, Vaibhav

I'm very interested to try your memcache extension in my qcubed-based project.
We have faced with serious performance problems on a customer side and looking for approache to optimize our product.

we have both 1.1.3 and 2.0.2 based parts. the problematic one is based on 1.1.3 version of qcubed

please, provide me your patch or a set of modified files to test impact on performance on a real-word application. I'm planning to make xdebug performance logs with your patch applied with and without memcached enabled and compare results

olegabr,

The work I have done is not complete and I would not recommend it for a live system (I have read your complaints about the poor performance on the other thread). I too am worried with the performance and hence tried it. Moreover I do not have version 1.x and I worked only with version 2.0.2. I cannot promise anything more about version compatibility.

I am still trying to add new functionality based on memcache. Also, it will need testing. I would love to have your feedback on it. As of now, I am deep with work and would not be able to provide the modified files (I do not really know how to patch.. it always fails for me) for at least today.

However I must also say that I have not improved anything in the querying aspect and all that I have changed are 'Load' and 'Delete' methods of single objects (since that is what was needed on my end).

Please wait for a few hours and I should be able to send you the files I have modified.

Regards,
Vaibhav

ok, thank you, Vaibhav for your reply.

I would wait for your files. I have paid programmers and testers to play with it and, probably iron and share it for community.

@olegabr: You can help me test and develop the memcache extention I am planning to. My database is not huge and overly complicated and there is no way I would be testing all the functionality in QCubed. Please tell me what is that is most required out of a caching solution (that would help me realize what I might be needing in future, as the application grows). I would like to work on it . Please do tell me when you have a test environment ready.

I am going out for dinner now, will come back and create a ticket which would do the job.

Created the ticket. You can download the files and start testing.

http://trac.qcu.be/projects/qcubed/ticket/770

I am not getting any response from the project developers here. Does idea sound too bad and I should not be doing this? :(

I need someone help me test the functionality.

BTW, I made a few new changes to the templates and other pieces of code which ensure that you can enable and disable Memcache support for:

1. The complete QCubed installation: The master variable USE_MEMCACHE in configuration.inc.php ensures this. This ensures that in case something is not going well with memcached (such as the server did not start up or, a memcache node went down and you are recieveing errors when trying to add it to memcache object's server list), you can simply switch off the use of memcached.
2. Individual databases (ORM Classes): A new case "case 'Memcached':" in the __get method of QDatabaseBase class in QDatabaseBase.class.php and a new index in the database configurations in configuration.inc.php "'memcached' => 'true',". The getter function will simply return the value of the 'memcached' array key in the configuration file (adjusted QApplicationBase's InitializeDatabaseConnections function to for the same). The rest of the code for database level memcaching expects the value of this key to be the string 'true'. This, altogether allows you to enable or disable caching on all tables in the database. This is immensely useful for those databases which contain sensitive data and you do not want anything left in memcache. Memcache in its normal usage does not use an authentication method and hence, anyone can connect to the memcache server and get the data from memcached's cache. God bless firewalls, but if someone was able to peep into your server somehow, this would be more troublesome because here, unlike databases, there is simply no password protection(until memcached added support for it recently).
3. Individual tables (their ORM class): Additionally, all the 'Gen' classes for all tables in all databases will have a protected static variable:

   protected static $blnUseMemcache = true;

   This is by default set to true and does not change by default on re-codegen whether or not you have set its database's configuration to NOT use memcache. This is because all code will first check for the database level usage setting and only then will it check for the ORM class (table) level. This circumvents the need to add another complicated analysis algorithm during the Codegen run as well as allows you to control table level memcaching.

   NOTE: There is one problem here however: You cannot override the 'blnUseMemcache' variable of the ORM-Gen class (Such as LoginGen class) in the ORM Class (such as Login class). This is because

   1. PHP does not allow a derived class to override a static variable in the parent class. So the 'Parent::$blnUseMemcache = false' is not going to work on the ORM class.
   2. We cannot make this variable go to a lower level for that will break the functionality and make a lot of things worse.
   3. It is important that this variable be static (the Load function is static and you cannot use '$this' but 'self').
   4. We should not make this public to elevate some security.

   As such, this variable will be overwritten everytime you run codegen.

   So why table level caching? Well, an application might be created using a single database and there is possibility that only one of the tables in that database has some important information which you do not want anyone to read just like that (such as user password hashes). If someone is enabling memcaching on a database, then chances are less that he would disable the memcaching on too many tables of that database. A couple or so tables is what you do not want to enable memcaching on, usually. So go to the ORM-Gen class and turn off the variable! Yes this is a bit painful if you make changes to the model frequently, but you would be making those changes only during development. Just turn off the variables before deploying and you would be fine.

The behavior of codegened Load has been adjusted. I am working on other methods which I modified and should be able to complete them in some time and then will upload new files.

Regards,
Vaibhav

I've made some performance measurements for qcubed-memcached proposal on qcubed 1.1.3 based application

results are promising: with memcached support my qcubed-based web application works really faster: 23235824 vs 11229153 of integral cost per AJAX call, calculated by KCachegrind, or approximately 2 times faster!

here you can download three xdebug performance logs (sudo apt-get kcachegrind in ubuntu to install kcachegrind performance log viewer):
http://85.118.231.54/qcubed.memcached.profile.tgz

description:

71743359 2012-01-21 15:28 cachegrind.out.17654 - xdebug log from original version with memcached enabled but not really used
41380971 2012-01-21 15:50 cachegrind.out.7845 - xdebug log with Expand clauses commented out for request events datagrid
31586278 2012-01-21 15:50 cachegrind.out.8218 - xdebug log with Expand clauses commented out for request and operator events datagrid

some context:
request events datagrid: dtgRelRequestEvents originally has these Expand clauses:
$this->clsAdditionalClauses =
QQ::Clause(
QQ::OrderBy(
QQN::RelRequestEvent()->EventDate, false,
QQN::RelRequestEvent()->EventTime, false,
QQN::RelRequestEvent()->Request->Id, false,
QQN::RelRequestEvent()->Id
)
, QQ::Expand(QQN::RelRequestEvent()->EventType)
, QQ::Expand(QQN::RelRequestEvent()->Request)
, QQ::Expand(QQN::RelRequestEvent()->Request->Type)
, QQ::Expand(QQN::RelRequestEvent()->State)
, QQ::Expand(QQN::RelRequestEvent()->Service)
, QQ::Expand(QQN::RelRequestEvent()->Operator)
, QQ::Expand(QQN::RelRequestEvent()->Employee)
, QQ::Expand(QQN::RelRequestEvent()->RequestEmployee)
);
All of these clauses are needed because some datagrid columns needs data from tables expanded

operator events datagrid: dtgRelOperatorEvents originally has these Expand clauses:
$this->clsAdditionalClauses =
QQ::Clause(
QQ::Expand(QQN::RelOperatorEvent()->EventType)
, QQ::Expand(QQN::RelOperatorEvent()->Request)
, QQ::Expand(QQN::RelOperatorEvent()->Operator)
, QQ::Expand(QQN::RelOperatorEvent()->Employee)
, QQ::OrderBy(QQN::RelOperatorEvent()->Id, 'DESC')
);
page super.php has both of these datagrids with 100 rows per page, updated with rate of 1 time in 5 seconds

xdebug logs are for ajax updates only. initial page load is excluded from logging.

in order to obtain cachegrind.out.7845, I've commented out some Expand cluses in dtgRelRequestEvents to enforce Load() calls for some objects without any additional clauses (because current implementation of memcached support in qcubed needs no clauses in Load for objects to be cached):
$this->clsAdditionalClauses =
QQ::Clause(
QQ::OrderBy(
QQN::RelRequestEvent()->EventDate, false,
QQN::RelRequestEvent()->EventTime, false,
QQN::RelRequestEvent()->Request->Id, false,
QQN::RelRequestEvent()->Id
)
//, QQ::Expand(QQN::RelRequestEvent()->EventType)
, QQ::Expand(QQN::RelRequestEvent()->Request)
//, QQ::Expand(QQN::RelRequestEvent()->Request->Type)
//, QQ::Expand(QQN::RelRequestEvent()->State)
//, QQ::Expand(QQN::RelRequestEvent()->Service)
//, QQ::Expand(QQN::RelRequestEvent()->Operator)
//, QQ::Expand(QQN::RelRequestEvent()->Employee)
//, QQ::Expand(QQN::RelRequestEvent()->RequestEmployee)
);

in order to obtain cachegrind.out.8218, I've commented out some more Expand cluses in dtgRelOperatorEvents:
$this->clsAdditionalClauses =
QQ::Clause(
//QQ::Expand(QQN::RelOperatorEvent()->EventType)
QQ::Expand(QQN::RelOperatorEvent()->Request)
//, QQ::Expand(QQN::RelOperatorEvent()->Operator)
//, QQ::Expand(QQN::RelOperatorEvent()->Employee)
, QQ::OrderBy(QQN::RelOperatorEvent()->Id, 'DESC')
);

results are really promising. with memcached support this qcubed-based web application works really faster:
23235824 vs 11229153 of integral cost calculated by KCachegrind, or approximately 2 times faster!

I am really glad that you have got performance improvements with the extentions that I developed. However, I would like to know the query types that are run the most, such as Load / LoadAll / LoadbyXXX and so on.

This will help me develop the thing even more.

Regards,
Vaibhav

I have several kinds of data in my application, that should be treated differently while caching:

1. Static system data, like RelRequestEvent->EventType or Request->Type for events. These data never changes by software customer, only by developer
2. Configuration data, like Service, Operator or Employee. These data can be changed by our customers, but not often (once per week, for example)
3. Events data, like RelRequestEvent or RelOperatorEvent. These are exactly events, produced by system operators or by final customers. These data never changes after construction - no SQL UPDATE or DELETE is called on these, never.
4. Request data. In my application it is called Request. It is exactly a ticket that you get to obtain some service, that identificates your place in a queue of people waiting for the same service too. This data can be changed in any way. Request changes it's state from Opened to Called, then Complete, special manager can change the service request is called for, or time it is written on etc. It can not be DELETEd, but can be UPDATEd.

1. Static system data can be completely optimized with qcubed <typeTableIdentifier suffix="_type"/> feature in includes/configuration/codegen_settings.xml
2. Configuration data frequently is used to explain events or requests, so it is Expanded in queries for events, like in my profiling example. Vaibhav's approach to flush cache on UPDATE or DELETE is ideal for these data due to it's infriquently modified nature. But now we have no support for caching such an objects, if they are used in Expand queries of another big query for event objects. It is a problem, because one needs to modify existing application in order to get benefit from caching. Any insights from core developers?
3. Events data should be always cached. Instead of construct it from DB result, caching framework should just get it from a cache by it's PK.
4. Request data can not be cached at all. Caching framework needs to have a way to exclude some tables from caching.

In fact, QCubed now has caching framework QCache, that is based on filesystem. This cache can be used directly, or through a wrapper like QueryArrayCached (http://examples.qcu.be/assets/_core/php/examples/qcubed_query/qcache.php). But it not really useful, cause enforces very low level processing.

My opinion is that caching of DB objects should be a core part and not a user-level API like QueryArrayCached. Vaibhav's approach is a promise. I vote for something like QAbstractObjectCache, mentioned by vakopian (http://trac.qcu.be/projects/qcubed/ticket/770), that should replace QCache and be integrated in all ORM objects Load/Update/Delete/Insert functions, like Vaibhav do now with statements like:
if(USE_MEMCACHE == true)) { do something; }

QCache should migrate to QFileCache as it really is.
QNoCache should be introduced that simply does nothing, but always returns null in get().
Actual type of cache used, should be set by configuration constant, in the same way as __FORM_STATE_HANDLER__ is used to set QFormStateHandler type.

I'm ready to participate in this work, if approved by core developers.

Well, I was about to go and modify LoadAll and LoadByXX methods but then I went on to read quite a few things about memcached and I think that caching large objects is going to be real bad with memcache and also a couple of other problems are going to come in. I am dropping the idea of doing that.

In case one wants to cache a lot more, I think getting the database's own cache size is going to help a lot more. While studying the code of PostgreSQL 9.1 source code, I learnt a whole lot of things about databases (I actually focussed on the PG's executor algos) and I think its a lot better to increase the amount of memory the database can use rather than caching large data in memcache.

PG's cache behaves like an LRU cache (although it is not so in true sense) making it work quite similar to memcache (save the query processing that goes before and after the cache is read). Hence, I would not modify the other Array Load methods.

About the abstract Class, I think I want to know how that would be a good idea and how you and vakopian think about its implementation.

About the QCache thing, I agree, fully. Its not much of an use, specially because its file system based and the cache has to be reprocessed after being read in to memory. I think QCache should be used only for saving static controls.

NOTE:
In the previous post I talked about the blnUseMemcache variable. I dropped it because it is breaking the way QCubed behaves. I respect its ability to use custom methods in the ORM class over the ORM-Gen class. I think someone who does not want his table to be caches, should just override the load method of the concerned class. That would be much better for everyone.

Regards,
Vaibhav

Vaibhav, look at these pictures attached.

It is a QueryArray function for my request events ORM objects.
Look for QPostgreSqlDatabase->Query cost in non-optimized (cachegrind.out_.17654 [super.php].png) and in optimized (cachegrind.out_.8218 [super.php].png) versions. It is 364841 from 12338812, or 3%, and 245702 from 2482540, or 10% respectively.

By these calculations I want to show, that we are talking about different things. You are talking about caching results of database queries in order to reduce DB load, and, in this you are absolutely correct about database's own cache size. But! I'm talking about caching not DB query results, but qcubed ORM objects constructed after querying a database. Look one more on screenshots. Functions like InstantiateDbResult are the real qcubed bottleneck. In order to really improve qcubed performance we need to cache already constructed ORM objects, because ORM objects construction is extremely expensive in qcubed.

I hope, now you understand my needs in caching for qcubed. sorry my english and ask again, if it is not clear till now.

You have asked me about more Load functions that I need to be cached. It is QueryArray and QuerySingle. In my application there are almost none of direct calls for Load or LoadByXXX or LoadAll functions. QueryArray is used everywere, because in real application we rarely need all objects, or one specific object that can be identified just by additional clauses. We almost always need complex query like that:

And almost always there are some additional clauses used, like here:

Note QQ::OrderBy clause. It is used extremely frequently, but your current implementation forbids memcache optimization for any query with non-empty clauses, even if it is just QQ::OrderBy clause. So, please, explain what is a reason for

condition in your code?

Note also the QQ::Expand clause. It is used not less frequent than QQ::OrderBy clause. It is because in a real non-trivial projects database tends to be very interconnected with foreign keys (we use postgres). But now I don't now how to handle QQ::Expand clauses with caching enabled. In my profiling tests I've turned QQ::Expand off, but how to implement caching of ORM sub-objects, returned by complex query with a bunch of QQ::Expands? Ideally, I want to perform a complex query for my RelRequestEvent objects and after querying a database, qcubed should construct ORM objects not by InstantiateDbResult, but first try to get them from cache by it's ids. And, of cause, in a perfect world, DB query should be constructed in a such way, that it would not expand relations, if they are not needed by other WHERE clauses, but only used on a PHP side to show some information for user.

hope, it helps to improve the caching idea for qcubed

And, finally, about the abstract Class. Now generated code with caching support looks like this:

The idea is to rewrite it like that:

Note unconditional get and set on a QApplication::$objMemcache object. It is an instanse of some class that fulfills some simple interface, like one mentioned by vakopian here http://trac.qcu.be/projects/qcubed/ticket/770.

We can implement a no-op cache class, like QObjectCacheNone, that would always returns null from get() and do nothing in set/delete/clear functions. This one would be used to disable caching:

Another possible values for this define would be QObjectCacheMemcache and QObjectCacheFile (QCache incarnation).

olegabr makes excelenet points here. I also commented on the ticket explaining why the abstract implementation and the `QObjectCacheNone` are needed.

Keeping the implementation backward compatible is paramount here. Asking the developers to override the Load() and other methods to just keep it working is definitely not acceptable. This is where QObjectCacheNone is important. Of course it's also important where caching is simply not available on the host.

While agreeing with olegabr on the need of handling caching for Expand and clauses, I also know that this is by far not easy. If you have ever used Hibernate you know that this is not a trivial subject. So I think we should start with the simple implementation that Vaibhav created, and think how to use the solution in other places.

@olegabr:

Yes, I got the point. We were indeed talking about two things - I was talking about caching the query results. As far as ORM objects are concerned, I agree that their construction is quite heavy. On a page which which renders 25 products on my site, I get a lag of approximately 5 seconds when first opening it. Of course the browser comes up, brings up the interface and all such lags are there. But once I simply stopped Apache and PostgreSQL. Then filled up the memory by loading all the programs I had (this will overwrite all the caches they hadin the RAM) and then closed them and started the webserver and database again. I still got a lag of 3 secs on the first run and approximately .6 seconds for subsequent runs. Worst was 1 second and minimum was .4 That meant that if I were to increase the number of things on the page, the page will get slow as hell. ORM construction is a bad part and yes, I am trying to solve the same here as well (along with the DB load); it's just that I did not talk about it till now but I am very convinced by you screenshots than I was before.

Now, about caching the ORM objects:

Queries depend on clauses. Reordering a datagrid by 'First Name' is different than reordering it in the 'Last Name' only because of the clauses (in this case ORDER BY). Now, If you were to take the query and use it as a key to memcached, you would be fired at by memcached for it does not accept keys any longer than 250 characters. So I had indeed taking the $objOptionalCaluses and run an 'MD5' over it to make it go down to 32 characters flat, no matter how huge it was. Change the Query, the clause changes and you get the respective data from memcache. Use the one you used before and you would be retrieving the data from memcached, not from DB because it would have been saved in the memory during the last request ran. It was easy. But... there are a few problems:

1.
Memcached takes a little longer to figure out the large key names than with short ones. Well, that is negligible. So we can ignore that.

2.
Memcache does not allow you to store more than 1 MB of data for single key. Ok, we can increase that with -I 4m when starting memcached which would actually ask memcache to set the upper limit of size per object as 4 MB. So it depends on how much data your queries produce and how often do you change the clauses, because as you go on changing the clauses, new values are fed into memcached (for every new clause, a new query is run and stores the data in the memory). Since in my application as well, the clauses do not change TOO much, this could be a viable option to look into.

How ever this can be problematic when too many large queries run, it will eat more space and memcache will expire the smaller objects which is again going to shoot the DB load up and when both of them are runnign frequently, well, all would depend on the size of the RAM.

3.
We already know there is no way you can find and delete keys from memcached (or for any other caching solutions , both persistent and non-persistent). So, there is a scenario which actually kills a few good with memcaching:

Let us say you have 100 single/individual ORM object cache in the memory. You also have a ORM cache of a huge and slow query which contains 200 items. Among those 200 items, you have one item whose ORM cache was also individually cached (one of those 100 items). Now, you delete that 'common' object from the database. Well and fine untill you find that the 'huge' query results you are dumping on the screen contains the data of that object which was deleted (its value was deleted on the DB as well the cache was deleted) but since the cache of the slow query was not expired, it still shows.

That scenario is a problem where memcaching can suck you up. The solution, to the best of my knowledge is to set an 'auto expiration' time in seconds when creating the cache. Let us say we set it to '120', making it expire automatically after every 2 minutes. Now, assuming that the slow query and the ORM object creation in total take about 10 seconds to complete, you would have a '10-seconds' LAG every 2 minutes. Look closely and you can be a lot slower as well if some thing like this happens:

1. You are getting 1 request per second.
2. The query takes 10 seconds to run.
3. A request makes the query run and results get cached into memcached.
4. Result expires at time - 11:00:00 exact.
5. At 11:00:01 , a new request comes which asks the database to re-run the query for the cache was expired.
6. At 11:00:02 another query comed and DB gets slower.
7. At 11:00:03, yet another query comes and DB gets even slower (this is because the query is not complete yet and the result is not yet cached).
8. Due to increased load, the query takes 20 seconds instead of 10 to finish (in PG code, there is a facility wherein, if a query comes which is already in execution, the results are sent from the same result source, but to decide that takes some time. I am not sure about MySQL here).
9. Your users are pulling their hairs and have hit reload multiple times and clicked a lot of other links on the page in frustation.
10: You DB curses you for the torture.

That is the whole scenario. The key points here are:

1.
How much data are you going to fetch in one go. Do you have enough RAM for memcached?

2.
How frequently is the query run?

3.
How frequently would you delete an individual item separately which was already cached in a set of other items?

4.
How 'LIVE' you want to make your website appear - when you delete items from the cache, they should simply reflect everywhere? This creatainly depends on how frequently you delete things.

I think that yes, auto expiration should be allowed for ORMs of slow queries and that users should be really aware of what is being done. I would like to hear more.

Regards,

Vaibhav

"Now, about caching the ORM objects". You are talking about caching the whole query result, using some hash (md5) as a key for the query cached, isn't it?

My vision is about caching not whole query result (with many objects and subobjects if QQ::Expand()s are given), but every individual ORM object from the result set, identifying it by it's own Primary Key (PK).

I was really impressed by profiling results obtained with your qcubed memcached extension. And it was exactly caching of individual query subobjects. That is why I'm thinking of caching not whole query result, but per object and subobject.

Per object caching has the advantage of reusing query results of subobjects from very different queries (both queries can QQ::Expand the same object, that would be obtained from cache for every not first query, even for very different one)

I have not catched your point about expiration time for ORM objects. Why it is needed at a first place, if you can clear cache for affected DB object on UPDATE/DELETE statements? Or, may be per object caching strategy solves the problem you've mentioned?

Per Object caching solves the problem, yes. But there are a few hiccups there:

If there are more than one primary key columns, we are probably going to get confused about how to do that. Let me have a look deeper in the code and I will get back. I think you have a nice point but that one takes the nerves on me if I try to do that! Let me try.

Plus, yes, I am going to change the Cache API. I spent hours looking at the options mentioned by Vakopian in the ticket and I think Reditt does a great job in caching too. It can also cache larger (much larger) objects (in fact as big as 2 Gigs) so that it can be a viable replacement for memcache in case someone is trying to use QCubed with a table which stores a BLOB or other Binary data type - e.g. I once thought of serving images from the DB as blobs.

Regards,
Vaibhav

OK. So I think I have completed it and QCubed can now cache every row in any query but I think you guys will have to test it thoroughly, both for correctness as well as for performance. Also, the API is not changed to the Get, Set and Delete function calls. I will upload the patch shortly.

Indeed caching individual ORM objects based on their id is the right approach. To cache query results, we can cache the list of ids of the resulting objects. That's what hibernate does. Of course we still have to be creative as to what to do with Expand's.
If memcache has such drastic limitations, caching query results with memcache would become problematic very quickly. Yet another reason for allowing for different back-ends.

Done with caching. Uploaded the patch (ticket770.patch) as well as supporting library (predis) for Redis support.

See here: http://trac.qcu.be/projects/qcubed/ticket/770#comment:8

Testing is required now.

I have done all that I could. I will tell what all I have done, its implications and negative points (and why I think they will be there):

I have added caching support for two backends: memcaches and redis.

Now, for that, I created a new class called QMCacher. I wanted to call it QCache but I can't, its already there. Plus, it will help those who are already there with QCubed since some time. The QMCacher class is there only to make use of other two classes namely QMemcachedCacher and QRedisCacher who are in turn derived from the abstract class QMCacherBase class. Creation of QMCacher makes sure that you can change the cacher in use just by changing the variable on fly in configuration.inc.php. No code changes are required. It also makes sure that in future, when new backends are considered, they will be added painlessly.

The API in the generated classes will change accordingly (templates are the reason for that).

Please see my new comments on the ticket: http://trac.qcu.be/projects/qcubed/ticket/770#comment:9

I have done all that I could. I will tell what all I have done, its implications and negative points (and why I think they will be there):

I have added caching support for two backends: memcaches and redis.

Now, for that, I created a new class called QMCacher. I wanted to call it QCache but I can't, its already there. Plus, it will help those who are already there with QCubed since some time. The QMCacher class is there only to make use of other two classes namely QMemcachedCacher and QRedisCacher who are in turn derived from the abstract class QMCacherBase class. Creation of QMCacher makes sure that you can change the cacher in use just by changing the variable on fly in configuration.inc.php. No code changes are required. It also makes sure that in future, when new backends are considered, they will be added painlessly.

The API in the generated classes will change accordingly (templates are the reason for that).

As for the negative points: I do not know about other drivers but pg_fetch_array() and mysql_fetch_array() both return the rows with both numerical as well as associative indexes. I am not sure about others. If the database row retrieved for the query does not contain numerical indexes, caching will simply not work on those databases. Please see if they do, or else we would need to alter the GetColumn function for those databases to help do that (check if the strColumnName parameter is numerical. If yes, call another function which can get the correct row).

Also, for caching to work properly, the table must have the first column as its 'single-column primary key'. If no, caching does not work. Such format is a mandate for caching to come to work.

Now, while working for 18 hours flat regularly, i found a couple of things about the 'expansion' mechanism: The first thing is: you cannot and should not try to cache fully expanded objects. If a person creates a really complex structure for this DB, then wanting fully expanded objects will ask for trouble:

1. You must do FULL EXPANSION to create such an ORM object. There is no way you can create that while the query is running and it DOES NOT specify all expandable fields to be expanded.
2. Also, it is really complicated to be able to cache fully expanded objects while the expansion is still going on. The amount of processing for such a process will take long enough to kill the benefits of caching.

Hence, only new formed rows get into the cache. Also, the unnecessary related objects remain as 'NULL'. This makes sure that the next time the related objects are asked for, they are fetched from the database and put into the memory for later use.

I also implemented Redis in becuase Redis does not start behaving abnormal with large amounts of data on single key. In fact, it supports up to 2GB of data per key. Also, it can be made use of as a database essentially making it an on-disk cache which would still be faster than running a complex query which takes seconds.

Please test it as much as possible. I have done my best and found it working great. In case you get problems, tell me and I willl try to correct my mistakes.

Regards,
Vaibhav

I replied there. And sorry if you feel bad after reading it. I was a bit angry while writing.

http://trac.qcu.be/projects/qcubed/ticket/770#comment:10

I found multiple problems with caching and it turned out that the only way to do it was by merging my and vako's methods. I did it and have moved some of the load from the runtime to code generation time.

I removed the GetObject, SetObject and DeleteObject functiond and their need. Instead I moved the whole code of what it does to codegen and the table class generated would automatically do all that those classes did - No loss of functionality, performance improvement!

Added a new variable for the 'delimiter' in the configuration.inc.php. During codegen, it takes effect and get replaced everywhere (this is one of the main reasons I could remove the need of those GetObject? / SetObject? and DeleteObject? functions).

Dropped support for redis. Till now, I was testing with plain and simple scripts with QCubed and it was going fine. When I tried using the code in the site I am developing, things broke down. There is no php library good enough to allow Redis to store PHP object. All it could store was the Object name like "User Object 3" or "Resource Object 7". I tried serializing and jsonencoding and nothing works there. Redis support has to be postponed untill there is a good enough library. The error I get is: "Trying to access property of a non-object". May be I was dumb to not try till now during all the testing!

The way I was trying to modify InstantiateDbRow? had two problems: my method was not allowing more than one primary key, while vako's did. I utilized his method and things work pretty fine. Also, for some reason (that I could not figure out), I was getting wrong results for Query Expansion when trying to set the object cache at the bottom of the function. I moved it to the place where the new object is created. I tried placing the code at different positions and either it would produce errors with PHP or the results that should be there (with errors). So I moved it. QCubed cannot make use of Query Expansion (read object explansion) with live caching, at least as of now. For storing 'BIGGER' results, someone might want to use QCache object. It works well. I did not implement something like that to avoid overloading memcache with huge amount of data coz it does not do much good in that case.

That is all for now.

@olegabr: Please see the patch ticket770.patch in the ticket and tell us how better is the performance. I do not have enough data with me to test and I tried to do performance profiling with Xdebug but could not - it was my first time I tried and XDebug has a lot of options; if possible teach me the trick :P

Hope it helps.

I've played a lot with previous version of your caching implementation. There are a preliminary results:
0) kcachegrind shows that time spent for data load reduced by factor 3.6!!! (524140 vs 1912179). It is because new version can cache QueryArray results, and most data like rel_request_event objects in my application is loaded with non-trivial QQuery by QueryArray function.
1) I've found that adding local cache to memcache is great (successive Get() from local cache costs 11 vs 510 for memcached):

This addition reduces interprocess memcache calls from 1100 no 320 for one AJAX update call in my application. Moreover, in some cases such a local cacher can be used instead of a memcached one.

2) in class_load_and_count_methods.tpl in function InstantiateDbRow you use statement (USE_MEMCACHE != false) && (QApplication::$Database[<%= $objCodeGen->DatabaseIndex; %>]->Caching == 'true') twice. I've factored it out to a variable and reused for second time (Caching is a property and you know my problems with it. And yes, kcachegrind shows me that this double property call is costly enough)
$useCaching = (USE_MEMCACHE != false) && (QApplication::$Database[<%= $objCodeGen->DatabaseIndex; %>]->Caching == 'true');
GetColumn is not for free too, so I use $cacheKey = "<%= $objTable->ClassName %>:Obj:" . $objDbRow->GetColumn(0); to call it just once.

3) It is not clear for me why you do the (!$arrPreviousItems) && (!$strExpandAsArrayNodes) check. I've commented it out and it seems to work. I've not tested it much, but in a first sight nothing crashed after this check was turned off. So, what this check is for?

I have more then one databases from different customers and test caching with it. Now I've got a frustration because after switching to another database I've got some objects from new database and most of objects from the old one.

My proposal is to include a database name into the cache key. Also, it would solve the problem with qcubed applications using two or more databases, that has tables with the same names like "service" in my case.

I did that on purpose. Why don't you do this: add a 'prefix' (or a suffix) to the database table names in the codegen.xml settings file for the second database. This will automatically make all the table names from the other Database unique and old objects will not trouble you!

Moreover, I had said that I did test things as much as possible. What I found is: If you have two table with same name in different databases, QCubed does not overwrite the table class for the 2nd (or 3rd or Nth) database. It simply gives out a warning. Prefixing and suffixing is for the same.

I have provided updates and quite a few things have changed. Some load has been shifted from runtime to code generation time. As for your things:

1. I am happy that things are working.

2. Adding Local cache is great. But let us say you never call 'delete' functions on memcache. How do you make sure that your 'LocalCacher' does not overrun your memory or does not starts swapping to disk? There seems to be no check. Memcache is a LRU cache and in case when data allocated in RAM is full, it will start expiring the old objects with LRU algorithm. Your LocalCacher does not do that. So if you have HUGE database and pretty less amount of RAM, fetching more and more data will bloat out memory allocated to PHP. Isn't it?

Yes, the benefits are great and for once I did think about doing that too for saying that QCubed would come with its own 'memory caching' system but I do not know a way to make sure that a particular variable (or array) in PHP does not get beyond a certain limit of RAM. Do you know any such mechanism?

Another problem: You probably tested the performance with a single script and a for loop. That is good. But with normal operations, when a PHP page ends rendering, all the memory allocations done by it are eligible for being cleared, which means that you cannot make the 'LocalCacher' object live accross page calls, forget user sessions. Since your tests would have ran all in a single page, PHP did not clear anything on that object. But it has no chance of survival accross pages. Memcached is a separate server with no direct relation to PHP and does not care which PHP page is requesting which Item, hence it works.

[ I AM ACTUALLY AMAZED THAT YOU SAID THAT IT WORKS FOR AJAX CALLS! HOW? ]

3. I cleared that in the latest patch I believe. Also removed call redirection being done by the previous QMCache. Should improve performance.

4. The reason for those additional checks is that I got wrong results without them. I inspected and found that it is something like this:

Let us say that there are 3 tables in the DB (taking from the examples site): Person, Login and Project. The person table has foreign keys to Login and Project so it is possible that you can expand a 'person' object for either 'Login' or 'Project' or Both. Consider the following scenario assuming each table has only one row with each row having primary key of value '1'(for simplicity).

a) A query is issued which asks for Person number 1 (no expanded items).

b) InstantiateDbRow kicks in, does not find the person object in memory. It loads it and saves [Person:1] to cache.

c) A second query comes asking for Person number 1 expanded to 'Login' property. The InstantiateDbRow (or 'Load' function, if that was called) finds the [Person:1] in cache and returns the object with no expansions.

I hope you can now see the reason why they are in place!! There is no way fast enough to make out what properties were expanded and what were not. If the cache is set, the object is simply returned without checking.

That check ensures: "If we are in a process of expansion, please do not get object from the cache".

This happens for every table class. So what you get is essentially single, non expanded objects into the memory. InstantiateDbRow's behavior is unpredictable - it depends on the Expand clauses. The check ensures that you get your objects as they should be.

1. consider two DBs with the same structure but different data, named DB1 and DB2
2. table DB1.T has one row with id=1, and table DB2.T has one row with id=1
3. firstly, I'm connecting my qcubed application to DB1 and see one record from table T. It is cached with index [T:1].
4. I'm switching DB to DB2 and request returns me one record with id=1. I'm trying to get it from cache and voila! I found object with index [T:1] from DB1

I have not tested the case with two DB in the same application. Of cause I use prefix to codegenerate it in my application.

1. local cache is essentially local. it's lifetime is exactly during just one server connection - while page loads or while ajax callback is performed. it is by design and by intension. consider the following data storage levels:
1) database with persistent storage. stores data always, regardless of connection or service/server restart/reboot
2) in memory cache server like memcached. stores data only till service restart or server reboot, and can drop data to make a place for new one. NOTE: it stores data between subsequent page reloads or ajax callbacks.
3) local in memory cache. stores data only during page load or ajax callback is being processed. does not stores data between page reloads or ajax callbacks. by design. by intension.

These three levels are clear and natural. I've introduced the last one because it really helps to save time in my application - it significantly reduces calls amount to the external persistent (between page reloads) in-memory storage like memcache. I have a great amount of objects that can be cached locally in the third level of data storage.

You've mentioned a possible memory exhausting problem with such a simple implementation. Yes, you are generally right. My opinion is that it highly depends of a particular application. In most cases where you should show tens or hundreds of records per page for your customer it should be enough.

May be it worth to make the local cache type an option to be set in configuration.inc.php? If one is not satisfied, he always can wrote more sofisticated one with memory limits management.

2. About that condition:

I've found that some of my code doesn't works as expected if this condition is commented out. For example, things like $objSvc->_RelServiceGroupsArray, obtained from QQ::ExpandAsArray(QQN::RelRegistrationService()->Service->RelServiceGroups) expansion are empty. It is a reverse relation expansion case. Reason is clear now. If that condition is not checked, all complex expands of reversed relations or virtual attributes are just skipped in InstantiateDbRow function.

I've found in my generated class for Service table that fields for simple expands and for reverse or virtual ones are processed in different ways: there are a check and Load call for simple ones and simple return statement for complex:

If we do use a per object caching, and I think we do, than we should drop the initial condition:

and unificate properties access rules for ORM objects. Complex expansion results should be handled exactly as simple ones. In this case one can utilize caching for complex queries and without a risk to broke code like $objSvc->_RelServiceGroupsArray. My point is that in a case of caching, this code should just perform a query in the same way as it is done for Timetable in the example above.

vakopian and alex, what do you think about this qcubed core change proposed? Does it brokes some architectural ideas that I'm not aware of?

Why it is important in a first place? With a condition mentioned:

we have no caching support for QueryArray function. But it is a working horse in my project. I can benefit from caching by commenting out my Expand clauses and obtain subobjects from a cache, but main application objects like RelRequestEvent, queried by QueryArray function are out of that optimization process completely. Please, respond.

I tried codegeneration in that case and I got two different filenames and classnames, each with the prefix added. Now, If you notice, the caching code generation algo uses the 'ClassName' for which it is being generated. When you use prefix for the tables, the class name automatically changes and hence, the name of the object in the cache will change. I do not understand how you are getting same names in the cache when using prefix!!!

I got your point about the local cache - its only for the 'Ajax' calls! I thought you were trying to use it as memcache or something. The way you have described it - its a really good idea. I think this can be done but then, if it persists even after initial form render is complete, it must be getting added to the 'FormState'. In that case, your formstate data is growing bigger. At times, this can be problem...but rarely. I do not know what to say. I am 50-50 for the local cache thing.

For the query Expansion -> I had tried what you have said (about removing the condition) as the first thing when trying to modify InstantiateDbRow and I had got weird results. While first expand queries ran well, others did not. In addition, if you remove a reverse-related object from table and then run the query with expands on the table, you still get the expanded reverse relation object in your result (which should not happen). Yes, on a normal day, that would not happen because when you drop the reverse object from the DB, the current object can also get deleted. But that will happen only on ON DELETE CASCADE. Plus, QCubed supports MySQL which can actually allow you to change table types. In case all tables are MyIASM tables, there is a file which can be used in QCubed to map table relations.

Such cases break if you cache expanded objects in memory. Moreover, I got wrong results and I would not be trying to remove it. From my side, caching stuff is over. I am done with it. It works all perfect. I am getting 80% boost on a normal page load. My pages which loaded in 830 ms approx are being loaded in 500 ms. yes, its not great but most of the load in there is because of creation of the controls and many calulations in the construct and render function of those controls. I do not know how to do performance profiling (someone guide me to the door, and I can run out easily) but I am sure, the benefit would have been manifolds. Removing those checks from both the 'set cache' point and 'get from cache' point results in erroneous results for me. It screwed up the Product display system badly. I am not in favor.

Again, about the 'Local Cache' thing, I think the core devs are the decision makers here.

[EDIT: I am making an all-'navigable' website. Something like blog. Things which are to be dynamic without being reloaded are very small ones and I control them using JS actions. So I do not know too much about ajax in this case. There is pretty minimal use to me for that. Also, my app breaks down approximataly 80% if I convert ServerAction to AjaxAction (because I do need Page reloads for some things to take place without being a memory hog). So I cannot say a word about Ajax.]

Regards,
Vaibhav

I agree with olegabr that the database name should be part of the cache key name. But that's pretty easy to do by getting the DB name in the codegen with "$this->objDb->Database" and including that as part of the key.
However I think there are more problems with the current implementation related to e.g. virtual attributes (see http://examples.qcu.be/assets/_core/php/examples/more_codegen/virtual_at... ). If you first load a simple object (with no virtual attributes) it will be cached. Then if you load the same object with some virtual attributes included, then the object will be loaded from cache and thus will not have any of the new virtual attributes.
I'm sure there are more problems like this that we are currently overlooking. And since we're trying to release 2.1 soon, I don't think we have enough time to workaround all these issues. So I think this feature will probably be pushed into a future release.

I overlooked that. And your comment on the ticket is also valid. This is THE reason I did not do caching with 'InstantiateDbRow' function in the beginning (although I actually was worried about the reverse relationships, I did not remember this one!).

I would say, let us keep it as is and just drop the caching code from the InstantiateDbRow for now and allow caching with the FIRST implementation => caching works for 'Load' call with no optional clauses - Will make the job easy to some extent and push the feature into 2.1 release. We can later continue on this feature, and develop it more for 2.2 release with 'improved caching' tag!

There are many people who would just use 'Load' calls. My application heavily uses it! Also, all the 'non-virtual' attributes of reverse relationships are handled by getters who call Load* method(s). So for single objects cached with 'Load' function will not cause problems. I am in support for that! (that was my 'first' idea as well!)

Please see my comment on the ticket http://trac.qcu.be/projects/qcubed/ticket/770#comment:27

I just uploaded my new patch into the ticket. This includes olegabr's suggestions about using the database name as part of the key and his local memory cache. It also include a QMultiLevelCacheProvider that would allow using the local memory cache together with other caches (or any combination of caches).
Setting the cache is now moved to the Load() method as discussed.

olegabr, I think your other ideas about restructuring the InstantiateDbRow method would have to be considered only after the next release, otherwise 2.1 would be delaied way too much.

I need some (actually a lot) of help in understanding the querying system of QCubed. I am planning (and trying to figure out) something which will make it stand out so very apart from all the others and I mean it. Obviously, it is my need and so I wanna build it into QCubed.

If you can put off the 'virtual binding' feature, the thing I am planning to make would just get fine into whole of QCubed. As of now, there are very few people using QCubed and almost no one prefers it for web development. Lack of documentation has earned QCubed as one of the 'worst' documented projects ever from all my friends to whom I recommended. I already created a few Wiki Pages which related by tickets. Please at least try to create some documentation there and that will help a lot!

The thing I am making would not only give QCubed a significant feature but also catch attention of many more people. The only trouble I am finding is virtual binding. :(

Wanna know what I am planing to make? ;)

I've attached a diff of my changes to Vaibhav's memcached implementation. some points can be used to improve current vakopian's implementation for the upcoming qcubed release.

1) caching for LoadByXXX functions added, it is used in my code and pointed to by the profiler
2) Save() function improved to call cache->delete only if SQL UPDATE was actually called. I've an error from memcached that key format is incorrect for newly created objects.
3) support for tables with only one column added (yes, it is crasy but I need it in my project to express a human concept that can not have nothing constant: no name or surname, age, male or whatsoever, all human's properties are placed in a different table and are a subject to change and it is important to track history of these changes)
4) in instantiation_methods.tpl I've turned on InstantiateDbRow caching for QueryArray with non-trivial conditions and clauses. It works only for non-virtual and non-array-expanded attributes (usage of results of QQ::ExpandAsArray expansion is banned for me). There were only one place in my code where result of QQ::ExpandAsArray expansion was used. I've dropped it out with two DB-queries and obtained a great optimization because of caching support for QueryArray everywere in my code. I understand that this part can not be accepted in qcubed core right now, but it can be useful for someone, I hope. Or, may be, in the meantime we can have a special setting in configuration.inc.php to turn on/off caching of QueryArray results, with a comment warning that it can not be used with array expansion or virtual attributes.

Hope it whould help to improve qcubed

A special thanks to Vaibhav for memcached support in qcubed. It is an invaluable addition, really. Everyone should try it.

With open source, things are born as the users start needing them. This was no difference. I too am enjoying the work I did and I would not take the caching scheme into my code right now, but only after 2.1 is finalized. I am happy with things as they are working now. Also, I will try to look into that "key format" problem when I am in 'development' mood.

Thanks for the appreciation. I too am enjoying caching and the 200% speed gain I got by simply changing the 'LoadBy' calls to simple queries and then performing 'Load' on the result. And yes, that is how the application is structured to run best on (LoadBy would not gain me this much performance) because my application has possibilities of a LOT of modifications on every single objects which gets loaded. So, I am sorry for not going to add your inventions in there. But yes, thanks for all the suggestions that you made. You helped me gain insights into the code generation mechanism up to an extent that I am ready for my next endeavor and that is going to make QCubed Unique, I promise (if done correctly).

Would any of the fine gentlemen here (obviously, including olegabr) would like to know what it is? :P