how do you define your schema?
how long does it take to insert/index/whatnot the data?
This is a much bigger deal than most people realize.
Poor schema design will cause your system to choke
bade when you try to scale it. In fact, relational
databases are not the ideal way to store this kind
of data so when you design your schema, you are really
fighting against the database to wrestle it into something
that will work.
this is a huge burden to figure it all out, implement
and then monitor/operate 24x7. miss enough samples or data
and you end up billing too little. this is why most folks
have either cooked their own, or use some expensive suite of
tools, leaving just a little bit of other stuff out there.
Personally, I doubt that it is possible to build a
workable system, even with plugins, that will do the
job for a significant percentage of service providers.
Different companies have different needs, different
hot button items, etc. This is an area where breaking
the problem down into well-defined separate problems
with a well-defined linkage, will go a long way.
But to start with, just solving the data storage problem
is a good place to start. If someone can create a specialized
network monitoring database that scales, then the rest of
the toolkit will be much easier to deal with. Note that
people have done a lot of research on this sort of
time-series database. People working in high-energy physics
also have to deal with massive sets of time-series data.
There is plenty of literature out there to help guide
a design effort. But Open-Source developers don't usually
do this kind of up-front research before starting coding.
Money and manpower won't solve that kind of problem.
--Michael Dillon
