Thanks. Going to look into this.

But brother, I need some help with the querying mechanism and some advice on the topic!

Basically I want to know how to create a new querying system with NoSQL (yes, I wanna generalize this). What properties can be there, which not (like some NoSQL support indexes, some don't and so on).

Regards,
Vaibhav

Ok, I understand. But don't be frustrated the goal is very ambitious and this is normal among open source that sometimes things can take a while until they gain traction.
So let us hope for a soon to come release of the new version of qcubed and probably then this can also be adressed.

Best,
tronics

I read now your latest post. This sounds awesome.
Here I already used MongoDB together with MySQL in the same project to have transaction savety and it worked great.
Love the idea to have the data in MySQL and querying it via MongoDB.

Why not use $dtt->format() (which will use DateTime::format())?

good point, I'll try it. but I have not only to format date but also use it in calculations, compare with current date-time, for example, and actually my code brokes because of comparisons of Timesatamps shifter by one hour. qcubed own datetime comparison function use timestamp internally, so they are broken also. it means that I can not compare non-serialized (QDateTime::Now()) and serialized times now. if it is not a bug, what it is?

olegabr,

I don't think there is a bug here. When you do
strftime("format", $dtt->Timestamp)
you are only using the timestamp information from the DateTime object, which knows nothing about the time zone.
Why not use $dtt->format() (which will use DateTime::format())?

From what I see in the code, the current serializtion of QDateTime correctly handles the timezone information since it's using the ISO8601 format to serialize.
With your change you are resetting it back to the default time zone, which may not be what the original object had.

the problem with timezone can be solved in this way:
includes/qcubed/_core/framework/QDateTime.class.php

Don't know how correct it is, but it works for me.

on a debian squeeze you need only
apt-get install php-apc
apache2ctl restart

and then, if you want to see apc statistics, read the result of:
cat /usr/share/doc/php-apc/README.Debian

Providing NoSQL support is as easy as creating a new class and adding new lines here and there in the code base. That is something Symfony project has done as well. As long as you have the drivers for the DB (in this case MongoDB) installed, things work (and this is no different than other DBs coz to make MySQL work with PHP, you DO NEED the php5-mysql driver).

The beauty of NoSQL databases is their ability to be unorganized and facilitate extreme flexibility - so you can have a row in the table (or 'document' in the 'collection') with 2 columns (or 'fields') or you can have 2000 or may be 20000? HBase officially declares itself as a Database for 'Millions of columns cross billions of rows'. That unorganized schemaless behavior with tremendous scaling and speed is the beauty of NoSQL.

The same makes it easy for the developers to make mistakes as well. For example, if you have created a collection called 'orders' where you store all the documents related to orders processed by your app. If you make a typo in any insertion call naming the 'orders' collection as just 'order' (notice the singular form), you are in hell of a mess because MongoDB ain't going to say "the collection does not exist'. Instead it creates a new one with the name 'order'. This is not the case with RDBMS systems which report back if you mis-spell a table name. Also, mis-spelling column (or 'field') names will create a new field. This is troublesome because we tend to make such mistakes and quite often (if you take away autocomplete, I would be effectively at least 3 times slower).

Now, the ORM approach utilizes the 'SCHEMA' of the RDBMS systems supported and NoSQL solutions such as MongoDB or Cassandra have no Schema at all! That seems to be your concern. I will tell you what my plan actually is. but before we try to make a solution, a revision of the scene is here:

==== RDBMS System =====

BENEFITS:

1. Transaction Safe - If the DB says that it's inserted, means that now even if there is a power failure, you data is recoverable.

2. Schema - Laws are laws and they are not meant to be broken in this world.

3. SQL - A universal way to query databases!

4. Widely accepted (do I really need to talk about this?)

DRAWBACKS:

1. Damn SLOW - The query processing takes a lot of time. Creating the right structure of commands internally to process data sucks at speed!

2. No Flexibility - If I wanna add another table or column... the procedure is hell tedious, slow and disruptive; you cannot think about it twice on production sites. Thinking anything as such for once itself was an error!

3. Horizontal Scaling sucks - If you want to go on to new servers, well, you need to sweat your fingers like hell.

4. Load balancing is much much more needed - Since these animals are slow, you need more cowboys looking after. define("AUTOMATION", NULL); If you plan to make anything which will be used more heavily than you initially thought, you are on the wrong side of the river.

That is the RDBMS scene and with all the bad things on its side, it wins with 'TRANSACTION SAFETY'; something without which critical applications cannot survive. RDBMS systems have to think about ACID (Atomicity, Consistency, Isolation, Durability). On the other hand we have NoSQL solutions which worry about CAP (Consistency, Availability and Partitioning [Network Partitioning] ). MongoDB is a C-P system with the 'A' achieved by distributing data onto multiple nodes which also helps in load balancing. Anyways, benefits of NoSQL are:

==== NOSQL systems ====

Benefits:

1. Very fast: Most NoSQL databases can beat RDBMS systems by orders of speed. MongoDB for one is approximately 4 times faster than MySQL and about 3 times faster than SQLite (take that!).

2. Horizontal Scaling: Most NoSQL systems have no problems scaling. Need example? Well, Cassandra and HBase are at production use inside Facebook! MongoDB can do the same as well! Here is the impressive list for Mongo: http://www.mongodb.org/display/DOCS/Production+Deployments

3. Automated Load Balancing: Most NoSQL systems either autonomously or with the help of additional parts built for the DB can perform auto load balancing. Cassandra is by default load balanced design (where in you can ask any question to any server in the farm and cassadra will reply back without bothering you) while HBase and MongoDB need other clients. MongoDB depends on the 'Config Server' for that (a part of the installation). Go on adding nodes to your farm (and configure them to run MondoDB at startup) and tell the config server the IP address of the machine. Forget the rest of work.

4. Schemaless: NoSQL simply is designed with 'Throw any data anywhere' approach. While it is better to keep related data under one collection (name for the virtual concept of Tables in MongoDB) or ColumnFamily (name for the virtual concept of Tables under cassandra), it is not a necessity. It is just that keeping related data under one collection would help you make indexes which improve the performance! Also, you can dynamically create a new collection and / or a column. This makes the system hell flexible. Also, you simply do not have to worry about 'NULL' values. If something is 'NULL', dont store it! Does not break any rules! So you can have everything under one collection as long as you know what you are doing and how you are going to manage. The DB will not be offended at all.

Being schemaless is the biggest plus point of NoSQL Databases.

DRAWBACKS:

1. No transactions: most NoSQL solutions are not safe for transactions. So if you have written something to the DB, you will have to wait for a significant amount of time until you be sure that the data is now recoverable and that a DB repairing will get you back the data. MongoDB provides a little support about this by allowing you to configure your language drivers to ask the server to confirm the write only after the change has propagated to 'n' number of servers. For example, If you have decided to make 10 copies of every data inserted, you can configure the database server to reply back to the client with the 'WRITE SUCCESSFUL' signal only after the data that was inserted has been copied to __at least__ 3 servers (or 4 or 5 or all 10 servers). That said, the load balancing is not synchronous and keeps going in the background without human intervention (just that you need to keep up sufficient storage). Transactions however are supported in terms of 'ATOMICITY' but if the node where the first and the only copy of the data was sent fails along with the master at the same time, there is no way you will be able to retrieve that poor chap!

2. Lack of support for multi-object transactions: MongoDB and most others do not provide too great a support for complex transactions which need to propagate changes to multiple objects in one go such as an 'UPDATE' statement which changes a few columns in a table for rows which were selected by a complex sub-select! NoSQL is not for such use cases. It is great with single objects however. If you want to introduce such complexity, do it with your programming code!

3. Use of SQL: Most systems today are programmed with SQL databases. If the working environment is in NEED of SQL, you cannot be helped with MongoDB or any NoSQL solution.

==============

That is all for the theory part. So the idea here is to make a merger of SQL's ability of transactions with the speed and flexibility of MongoDB. When there is just a few GBs of data but you are still getting more than 100 hits per second at peak time, there is no option other than to scale out. If the data is in TBs, nothing much changes if you make good decisions with Mongo. With RDBMS systems, that would be a great problem.

My idea revolves around the following design and workflow:

1. Native MongoDB support: A new class will be created which could support querying MongoDB.

2. Association of another DB's schema to a MongoDB instance seslected by user: Say I have a PG database named "blog", then I will have a corresponsing database in my selected (or associated) mongoDB instance too!

If I have two tables in the 'blog' database (say) 'posts' and 'comments', then I will have collections of same name under the mongoDB instance! Mapping of columns to MongoDB 'fields' would also be for each table mapped to a mongo collection. It is essentially mirroring the SQL database into the MongoDB's noSQL system (thus making it a little more organized).

3. Write Twice, read once: All the updates and inserts are sent to both the RDBMS (let us assume PostgreSQL) server as well as the associated MongoDB server. The writes are first sent to the RDBMS system and only when the transaction is reported complete (and we know it's now recoverable), we send the same data to MongoDB. When requesting data, it is first requested from MongoDB. So when extreme loads come in, MongoDB will be handling it all. Only if the MongoDB fails (which it should not if all servers were running fine and at least one copy of data was available), we get the data from the PostgreSQL system coz we know it is there for sure.

4. CodeGen: When running codegen for the PostgreSQL db with an 'associated' MongoDB instance, code for the querying and insertion of data into both is automatically created in ORM classes!

Now there are a couple of things which come into mind for troubles and I would take them (as percieved by me) one by one:

Q-1. Why try to catch the right ear with the left hand? Wasn't MongoDB support just enough?

Ans: Yes, it was. But making things this way will make sure that you do not make mistakes in 'names'. If you use QCubed's ORM classes, you sure are not mis-spelling the collection or field name. In case you are using custom queries, remember that RDBMS insertion takes place first. So you would be warned and insertion will fail anyway!

Q-2: Why two copies of data?

Ans: Its just your choice to have two copies! You can use MySQL with 'Blackhole' engine to make sure that MySQL system does not store anything!

Q-3: I dont want a separate installation of MySQL either. Can't I just have the MOngoDB support?

Ans: Of course you can. Just that when you are using MongoDB alone, it is completely schemaless and codegeneration can be done only on 'indexed' fields. All the rest must be queried the way it is done with the raw MongoDB driver. Association of another RDBMS is an option which prevents mistakes from happening. It is not going to be a necessity.

Q-4: MongoDB is schemaless and I can add any data anywhere. So why this SQL discipline over there?

Ans: As I already told, It is to prevent spelling mistakes (and a few other benefits). Moreover, all ORM classes generated for a RDBMS database instance with a Mongo-backing (that is what I call it) during codegen will recieve a special variable named _extra__data and will be accessible via the Extra_Data property. You can put any data into this field and use it to save to your MongoDB. Any data saved using this variable / property will not be saved into your RDBMS systems. Additionally another Database variable will be accessible to you for any other queries you want (e.g. QApplication::MongoDB[1]).

Q-5: So what are the benefits of doing this?

Ans:
1. You are effectively having the flexibility and speed of MongoDB for all the reads and writes as well as the transactional ability for your application with the RDBMS system having all the data as well.

2. Once you have made sure that Your MongoDB is now consistent and recoverable and has all the data as your RDBMS systems have, you can drop the data from your RDBMS or Mongo database (as per your needs). You can keep both copies of data as well. Anyways you would be having multiple copies in MongoDB, adding just another copy would not be so troublesome. If it is, decrease the load balancing factor on MongoDB by 1 and utilize the space!

3. MongoDB is extremely good for logging purposes. Usually Logs and performance and usage statistics are created as files and searching is a problem. Also, logs or statistics data are usually not a part of the core DB containing the data. You can use MongoDB to contain all the Logs (and they would be searchable!!!) and statistical data. With its speeds, you will not have a problem in searching and real-time analysis while keeping the actual database clean. You can also save 'per-row' data in every document corresponding the row in the RDBMS system or create a new collection or.... the possibilities are endless my dear watson!

4. If you serve multimedia from the DB, Mongo will make it faster by orders of magnitude with its internal load balancing ability.

5. You might just start feeling that memcached was 'SHIT' with mongoDB's 'capped collections' at work. It performs way faster than redis and is hardly 5-10% slower than memcached. Well, and it's distributed in the sense of 'load balancing' as well!

6. If you are serving videos from a DB, use good strategies and you will be saving a lot of processing and bandwidth with MongoDB.

7. Mongo is great for geospatial indexing. So you can use your RDBMS to keep the data but use a rather faster DB for querying it!

8. Unlike other noSQL Databases and Like RDBMS systems, you can create 'indexes' on your collection for certain fields. The difference from SQL world is that your indexes are distributed too! So, if the result for a query utilizing indexes was found in an index which was located on Server#237, the data will also be there. Less chatter on the network, faster!

9. MongoDB uses BSON (Binary JSON) as its storage format of data. This is particulary great for webapps. The mongoDB PHP driver takes the data input as nested arrays (with the same structure as the JSON object(s)). Just ask the data from Mongo, convert to JSON, shoot at the client browser to be handled by some JS! Saving data is no problem either. You can recieve data in JSON encoding, and directly send it to the DB. So your server load can go down drammatically low as a lot of processing can be shifted to the client browser!

Now, guys, how does it all look? :)

Regards,
Vaibhav

I tried Xcache n it did not work out well. APC screwed as well. I do not understand how to use them. If you happen to succeed, Please share it :(

Test P1.0: one member named 'intA': 0.418
Test P1.1: one property named 'a': 1.673
Test P1.2: getter function named 'getA()': 1.268

you are right, getter is slightly faster - __get is 4 times, and getter is just 3 times slower than direct member access

about this: