Has anyone used Monosphere in a volatile environment for prediction?

I'm curious because it looks good on paper, but I'm worried that the
predictive ability is only viable in a somewhat constant environment.

Thanks.

~F

[ Post a follow-up to this message ]

Hi,

what kinda predictions you are looking for ?

Is your environment volatile on the logical side of storage or is it
more on the connectivity/fabric part of it ?

Thanks
Harish

Faeandar wrote:
> Has anyone used Monosphere in a volatile environment for prediction?
>
> I'm curious because it looks good on paper, but I'm worried that the
> predictive ability is only viable in a somewhat constant environment.
>
> Thanks.
>
> ~F

[ Post a follow-up to this message ]

On 12 Jan 2007 06:45:54 -0800, harishlb@gmail.com wrote:

>Hi,
>
>what kinda predictions you are looking for ?
>
>Is your environment volatile on the logical side of storage or is it
>more on the connectivity/fabric part of it ?
>
>Thanks
>Harish

Sensible predictions.  I don't see how it's possible to predict 3
years out with 6 months worth of data.  Or any amount of historical
data actually.

The model that has fit best over the last 5 years is a linear log
growth.  But it does not predict "when" new disk will be needed,
simply that new disk will be needed.

Logical volatility, as evidenced by the linear log approach.

Thanks.

~F[vbcol=seagreen]
>
>Faeandar wrote: 

[ Post a follow-up to this message ]

Being an employee of MonoSphere I wanted to ensure that there was a
response to the request from Faeandar.  Given that the MonoSphere
Storage Horizon application is relatively new, I’m not sure that other
users would have the necessary background and experience to provide an
adequate response. So, with the understanding and caveat that I am an
interested party (and a strong proponent) let me provide some
information.

The heart of the Storage Horizon app is a very elaborate and complex
modeling engine. Storage Horizon’s Forecaster is capable of integrating
different forecasting engines in its core.  The forecasting engine we
currently use has a built-in expert system, allowing it to find the
best-fitting forecasting model out of a variety of
Exponential-Smoothing and ARIMA (aka Box-Jenkins) models.  It also is
capable of fitting a Simple Moving-Average model and other forecasting
algorithms. The engine, by itself, does not automatically detect
seasonal variation(s) or trends; nor does it automatically detect
events in the data, which we see a lot of in storage capacity planning
data. By passing the scenario through our proprietary
“DataPatternAnalysis” module prior to sending it to the engine, we
alleviate this concern, making our model “intelligent” enough to detect
periodic patterns, trend type, and events (outliers and level shifts).
We then interpolate outliers, making a note of it, and send data to the
forecasting engine with the additional information on seasonality and
trend type.  This symbiotic integration with the engine allows us to
every time obtain the best-fitted model as indicated by our
Model-quality index (MQI).

While we are more than capable of handling a multitude of growth (or
just change) patterns in the data; our experience, in various data
centers, has been that there are only a handful of data volumes that
exhibit highly random growth. Most neatly follow linear or exponential
patterns with some degree of seasonality. You are entirely correct in
stipulating that any attempt to forecast out farther than warranted by
historical data is wrong. While the application will allow the user to
create such a prediction it will also provide guidance and warnings
against this sort of action

Bottom line, we have been demonstrating extremely viable and useful
projections in all sorts of highly volatile environments. These sorts
of analysis have allowed users to gain pretty interesting and unique
insights into their environment. Please feel free to contact me
(bill@monosphere.com) for any additional information or a thorough
demonstration.

Thanks for opportunity to reply

wtmono

Faeandar Wrote:
> On 12 Jan 2007 06:45:54 -0800, harishlb@gmail.com wrote:
> -
> Hi,
>
> what kinda predictions you are looking for ?
>
> Is your environment volatile on the logical side of storage or is it
> more on the connectivity/fabric part of it ?
>
> Thanks
> Harish-
>
> Sensible predictions.  I don't see how it's possible to predict 3
> years out with 6 months worth of data.  Or any amount of historical
> data actually.
>
> The model that has fit best over the last 5 years is a linear log
> growth.  But it does not predict "when" new disk will be needed,
> simply that new disk will be needed.
>
> Logical volatility, as evidenced by the linear log approach.
>
> Thanks.
>
> ~F-
>
> Faeandar wrote:-
> Has anyone used Monosphere in a volatile environment for prediction?
>
> I'm curious because it looks good on paper, but I'm worried that the
> predictive ability is only viable in a somewhat constant environment.
>
> Thanks.
>
> ~F--




--
wtmono

[ Post a follow-up to this message ]

Faeandar wrote:
> Has anyone used Monosphere in a volatile environment for prediction?
>
> I'm curious because it looks good on paper, but I'm worried that the
> predictive ability is only viable in a somewhat constant environment.

The detailed response you just got from 'wtmono' made me curious as
well, given the effort they clearly went to in this area.

Just how much would a product (perhaps something like Isilon on
steroids) that allowed you to expand and reconfigure your storage
incrementally and inexpensively in a 'plug and play' manner reduce your
need for such predictive mechanisms?  I.e., (allowing for the need to
expand the LAN facilities for tying such a clustered storage system
together) how much of your need to predict is based on the effort
involved in coordinating the expansion rather than the raw equipment
cost of it?

- bill

[ Post a follow-up to this message ]

On Sun, 14 Jan 2007 16:39:15 -0500, Bill Todd <billtodd@metrocast.net>
wrote:

>Faeandar wrote: 
>
>The detailed response you just got from 'wtmono' made me curious as
>well, given the effort they clearly went to in this area.
>
>Just how much would a product (perhaps something like Isilon on
>steroids) that allowed you to expand and reconfigure your storage
>incrementally and inexpensively in a 'plug and play' manner reduce your
>need for such predictive mechanisms?  I.e., (allowing for the need to
>expand the LAN facilities for tying such a clustered storage system
>together) how much of your need to predict is based on the effort
>involved in coordinating the expansion rather than the raw equipment
>cost of it?
>
>- bill

I would say that depends on your environment.  Isilon is not cheap.
My guess is the true issue is a mix of prediction, easy expansion with
little to no coordination, and keeping growth under control.  Like I
said, my growth has been a linear log.  That is not something that can
continue for too long.  Eventually something has to give.  If you can
predict, or more importantly identify, data growth you have at least a
scant chance of flattening out the growth curve.

~F

[ Post a follow-up to this message ]

In article < DLadneSUQ_mZPjfYnZ2dnUVZ_uyknZ2d@metroca
stcablevision.com>,
Bill Todd  <billtodd@metrocast.net> wrote:
>Faeandar wrote: 
>
>The detailed response you just got from 'wtmono' made me curious as
>well, given the effort they clearly went to in this area.
>
>Just how much would a product (perhaps something like Isilon on
>steroids) that allowed you to expand and reconfigure your storage
>incrementally and inexpensively in a 'plug and play' manner reduce your
>need for such predictive mechanisms?  I.e., (allowing for the need to
>expand the LAN facilities for tying such a clustered storage system
>together) how much of your need to predict is based on the effort
>involved in coordinating the expansion rather than the raw equipment
>cost of it?

Imagine you had a cluster file system (which is goodness, because it
means that all your hosts can access all your storage, concurrently).
You then connect all your application hosts to all your storage
devices (whether they be SAN or NAS), and have all storage go through
that cluster file system.

Clearly, this cluster file system has to be heterogeneous.
Interoperability (and correct translation of file, access mode,
locking etc.) within Unix-style machines is comparatively easy.
Integrating Windows is harder, but mostly doable (with lots of work).
Integrating zOS (a.k.a. mainframe or VM and MVS) and VMS is outright
hard, mostly because they have different semantics for the data (CKD,
RECFM, RMS record formats, finally EBCDIC), so let's skip those.

Furthermore assume that the cluster file system supports full
security, consisting both of heterogeneous access control, and
normalization / translation of user authentication and user
identification.  This would require a policy engine to enhance the
security capabilities of the individual operating systems (for
example, the ACL semantics of Windows is richer than what is common on
Unixes, so you need policies to fill ACLs).  This implies that it is
actually safe to connect all your hosts to a single cluster file
system, as you can effectively separate them through security policies
if needed.

Furthermore assume that the cluster file system provides an
intelligent form of data placement.  This has to include policies
(such as any data from application X goes to disk array A, any data
from user U goes to NAS B, and so on).  It also has to include
policies that react to resource usage (a simple one is quota: stop
user U from writing if he is using too much space, much more elaborate
ones exist in SRM).  Also assume that the cluster file system provides
HSM and backup facilities (often called ILM or information life-cycle
management today, because HSM and backup are considered boring words),
for example by simply connecting to existing best-of-breed HSM and
backup tools.  Part of intelligent data placement is providing
transparent data migration, of placement decisions have made earlier
are no longer optimal.

In its placement policies, the cluster file system needs to take data
availability and data loss into account.  For example, if a policy
states that data is very important (like the e-mail of the CEO), it
needs to be replicated (preferably remotely), and placed on
ultra-reliable disk arrays.

If one had such a cluster file system (*), all of traditional storage
management (including SRM!) would become unneccessary.  Wouldn't that
be wonderful?

(*) The vision and architecture of this cluster file system actually
had a name, but that's a different topic.

--
The address in the header is invalid for obvious reasons. Please
reconstruct the address from the information below (look for _).
Ralph Becker-Szendy      _firstname_@lr_dot_los-gatos_dot_ca.us

[ Post a follow-up to this message ]

On Mon, 15 Jan 2007 04:23:00 -0000,
_firstname_@lr_dot_los-gatos_dot_ca.us wrote:

>In article < DLadneSUQ_mZPjfYnZ2dnUVZ_uyknZ2d@metroca
stcablevision.com>,
>Bill Todd  <billtodd@metrocast.net> wrote: 
>
>Imagine you had a cluster file system (which is goodness, because it
>means that all your hosts can access all your storage, concurrently).
>You then connect all your application hosts to all your storage
>devices (whether they be SAN or NAS), and have all storage go through
>that cluster file system.
>
>Clearly, this cluster file system has to be heterogeneous.
>Interoperability (and correct translation of file, access mode,
>locking etc.) within Unix-style machines is comparatively easy.
>Integrating Windows is harder, but mostly doable (with lots of work).
>Integrating zOS (a.k.a. mainframe or VM and MVS) and VMS is outright
>hard, mostly because they have different semantics for the data (CKD,
>RECFM, RMS record formats, finally EBCDIC), so let's skip those.
>
>Furthermore assume that the cluster file system supports full
>security, consisting both of heterogeneous access control, and
>normalization / translation of user authentication and user
>identification.  This would require a policy engine to enhance the
>security capabilities of the individual operating systems (for
>example, the ACL semantics of Windows is richer than what is common on
>Unixes, so you need policies to fill ACLs).  This implies that it is
>actually safe to connect all your hosts to a single cluster file
>system, as you can effectively separate them through security policies
>if needed.
>
>Furthermore assume that the cluster file system provides an
>intelligent form of data placement.  This has to include policies
>(such as any data from application X goes to disk array A, any data
>from user U goes to NAS B, and so on).  It also has to include
>policies that react to resource usage (a simple one is quota: stop
>user U from writing if he is using too much space, much more elaborate
>ones exist in SRM).  Also assume that the cluster file system provides
>HSM and backup facilities (often called ILM or information life-cycle
>management today, because HSM and backup are considered boring words),
>for example by simply connecting to existing best-of-breed HSM and
>backup tools.  Part of intelligent data placement is providing
>transparent data migration, of placement decisions have made earlier
>are no longer optimal.
>
>In its placement policies, the cluster file system needs to take data
>availability and data loss into account.  For example, if a policy
>states that data is very important (like the e-mail of the CEO), it
>needs to be replicated (preferably remotely), and placed on
>ultra-reliable disk arrays.
>
>If one had such a cluster file system (*), all of traditional storage
>management (including SRM!) would become unneccessary.  Wouldn't that
>be wonderful?
>
>(*) The vision and architecture of this cluster file system actually
>had a name, but that's a different topic.

Actually, I would argue that SRM would still be necessary as a tool in
a process to curb growth.  Just because I can move data all over God's
green earth does not mean I know what it is or it's value.  And since
business is primarily concerned with making money alot of cruft stays
around.
Theoretically that cruft would get moved to whatever tier you choose
but it would still be around.  In an ideal world it would not be
backed up anymore either.
Theory and optimism are both tough to swallow in data storage.

With regards to your side bar, I'm aware of a file system that will
transparently migrated data, HSM built in, policies, clusterable, etc,
that fit in a Unix environment (well, specifically Sun) and that's the
combo of SamFS and QFS.
The only file system I'm aware of that will do multi-OS clustering is
CxFS, but I can't say I'm up on the details of what it can do.

This is restricted to open systems of course.....

~F

[ Post a follow-up to this message ]

_firstname_@lr_dot_los-gatos_dot_ca.us wrote:

...

> Imagine you had a cluster file system (which is goodness, because it
> means that all your hosts can access all your storage, concurrently).

Coming from a DEC background, I used to believe that (and I think with
good reason).

However, circumstances have changed such that I don't believe it now.
In particular, the balance between the cost of storage and the cost of
the intelligence to serve it at the file level has swung so far toward
the former that making hosts deal with the complexities of cooperative
access to block-level devices (and requiring that they all trust each
other to do so in a timely and reliable manner) is somewhat hard to justify.

The best remaining argument for shared storage (rather than shared file
servers organized as a server cluster to distribute the access and
network workloads of however many hosts may be accessing them) may be
that a parity-redundancy group is kind of a pain to organize across
multiple servers.  If one doesn't buy into the idea that parity-based
redundancy is no longer cost/performance-effective in this era of
inexpensive disks, or believes that servers die frequently enough that
decoupling the storage behind them from such failures is important
(don't count server power-supply failures in that calculation, since the
redundant power that you need to guarantee that the shared storage sees
no single point of failure could do the same for the server), then
providing fail-over-style multi-initiator access to small storage groups
is a reasonable compromise that avoids the complexities (and overheads)
of concurrent disk-sharing (that was why I asked about SATA port
selectors here recently, but SAS is there if more flexible take-over
mechanisms are desired).

> You then connect all your application hosts to all your storage
> devices (whether they be SAN or NAS), and have all storage go through
> that cluster file system.
>
> Clearly, this cluster file system has to be heterogeneous.
> Interoperability (and correct translation of file, access mode,
> locking etc.) within Unix-style machines is comparatively easy.
> Integrating Windows is harder, but mostly doable (with lots of work).
> Integrating zOS (a.k.a. mainframe or VM and MVS) and VMS is outright
> hard, mostly because they have different semantics for the data (CKD,
> RECFM, RMS record formats, finally EBCDIC), so let's skip those.
>
> Furthermore assume that the cluster file system supports full
> security, consisting both of heterogeneous access control, and
> normalization / translation of user authentication and user
> identification.  This would require a policy engine to enhance the
> security capabilities of the individual operating systems (for
> example, the ACL semantics of Windows is richer than what is common on
> Unixes, so you need policies to fill ACLs).  This implies that it is
> actually safe to connect all your hosts to a single cluster file
> system, as you can effectively separate them through security policies
> if needed.

More good reasons for using file-level access from the hosts (via an
existing protocol that they already understand, though extensions
requiring enhanced host software could also be desirable - if Isilon
doesn't do this, someone similar does):  at least for CIFS/NFS access,
these problems are pretty well solved (my impression is that NetApp has
led the way here).

>
> Furthermore assume that the cluster file system provides an
> intelligent form of data placement.  This has to include policies
> (such as any data from application X goes to disk array A, any data
> from user U goes to NAS B, and so on).  It also has to include
> policies that react to resource usage (a simple one is quota: stop
> user U from writing if he is using too much space, much more elaborate
> ones exist in SRM).  Also assume that the cluster file system provides
> HSM and backup facilities (often called ILM or information life-cycle
> management today, because HSM and backup are considered boring words),
> for example by simply connecting to existing best-of-breed HSM and
> backup tools.  Part of intelligent data placement is providing
> transparent data migration, of placement decisions have made earlier
> are no longer optimal.

Why not just spread all the data across all the servers and their drives
(as Xiotech's Magnitude and then HP's EVA - and I'm sure by now others -
do at the block-access level, at least across the drives of a single
box) and use access-prioritization (rather than HSM and
storage-segregation by user or application) as the way to differentiate
quality of service (today's drives being large enough that they should
have loads of room for cold data if there are enough of them to provide
good service for the hot data)?  That certainly lets you get the most
out of your hardware (rather than leaving parts of it idle when their
dedicated applications don't happen to be very active, while others
elsewhere may be starved for IOPS or bandwidth), and at least reduces
even if it may not wholly eliminate the need to 'migrate' data, though
quotas and backup remain necessities.

Futzing around carefully placing (and then subsequently migrating) data
seems awfully tedious (and sub-optimal w.r.t. flexible use of resources,
as described above) if you can avoid it.

>
> In its placement policies, the cluster file system needs to take data
> availability and data loss into account.  For example, if a policy
> states that data is very important (like the e-mail of the CEO), it
> needs to be replicated (preferably remotely), and placed on
> ultra-reliable disk arrays.

Inheritable per-file specification of the type and degree of redundancy
would seem to cover that (especially if augmented by ZFS-style
end-to-end checksum verification) - without the need for specialized
arrays (I'm assuming now a log-protected file system into which
redundancy support has been integrated such that NVRAM is not required
to make it perform well - though a bit of NVRAM dedicated to holding the
log itself could make it *really* fly).

>
> If one had such a cluster file system (*), all of traditional storage
> management (including SRM!) would become unneccessary.

I've tried above to identify some residual traditional management
activities that might also be avoidable - and without many of the
complexities, overheads, and potential drawbacks (inter-client trust
being the real biggie) of a concurrently-shared-disk file system.

Wouldn't that
> be wonderful?
>
> (*) The vision and architecture of this cluster file system actually
> had a name, but that's a different topic.

Well, now that you've got the discussion started, don't hold back...

- bill

[ Post a follow-up to this message ]