Friday, March 30, 2018

How does AmiDb partition data for cassandra and elasticsearch?

I have created a model but I'm not sure how it stores data into cassandra or elasticsearch
My model (changed the field names for privacy) has ska (id="employeeid"), cola (firstname), colb (lastname)
Can you explain how the data is stored in cassandra or elasticsearch?
AmiDb creates partitions in cassandra based on the headers that you generate and the model name being used.
Hence data is always partitioned by the pscope (owner,[container,[tenant]])+modelname
So when you design your model keep this in mind to achieve horizontal scalability.
These partitions go to the keyspace the model is defined in, and models can share a keyspace.
AmiDb already creates keyspaces for you based on usage type, for example caches go to a separate keyspace, user information goes to another keyspace.
All the user data goes to one keyspace. In production we believe you may not need to create additional keyspaces, but if you desire so you can log into the management console and create this. (Not visible on free cluster)
For elastic search, since partitions==indexes and there is no keyspace, we can end up into too many idexes and open file limits if we were to do the same. Hence, amidb internally  collapses indexes into one and allows you to split or merge indexes based on usage pattern. One story under development is to automate this behind the scenes. For now you have management consoles to do this task. (Not visible on free cluster).
Your queries are always at the model level and hence they do not depend on keyspace or indexname changes.

No comments:

Post a Comment