|
Home > Archive > Snort > September 2004 > [Snort-users] Finding alerts taking up the most database space
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
[Snort-users] Finding alerts taking up the most database space
|
|
| McCash, John 2004-09-10, 5:46 pm |
| This is a multi-part message in MIME format.
---- =_NextPart_ST_11_20_48_Friday_September_
10_2004_13802
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi,
I currently am running snort and acid with mysql, and my
database size is getting up around 6G. The data table, data.MYD alone is
about 3.3G. As you may imagine, my db performance is lousy. Does anyone
have an easy way of determining which alerts are taking up the greatest
amount of db space, so that I can selectively prune those entries?
Thanks in advance
John McCash
---------------------------------------------------------------------------=
---------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information. =20
If you have received it in error, please notify the sender
immediately and delete the original. Any unauthorized use of
this email is prohibited.
---------------------------------------------------------------------------=
---------------------
[mf2]
---- =_NextPart_ST_11_20_48_Friday_September_
10_2004_13802
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3DContent-Type content=3D"text/html; charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=3DEN-US link=3Dblue vlink=3Dpurple>
<div class=3DSection1>
<p class=3DMsoNormal><font size=3D1 face=3DArial><span style=3D'font-size:8=
=2E0pt;
font-family:Arial'>Hi,<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D1 face=3DArial><span style=3D'font-size:8=
=2E0pt;
font-family:Arial'> &n=
bsp; I
currently am running snort and acid with mysql, and my database size is get=
ting
up around 6G. The data table, data.MYD alone is about 3.3G. As you may imag=
ine,
my db performance is lousy. Does anyone have an easy way of determining whi=
ch
alerts are taking up the greatest amount of db space, so that I can selecti=
vely
prune those entries?<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D1 face=3DArial><span style=3D'font-size:8=
=2E0pt;
font-family:Arial'> &n=
bsp; &nbs=
p; Thanks
in advance<o:p></o:p></span></font></p>
<p class=3DMsoNormal><font size=3D1 face=3DArial><span style=3D'font-size:8=
=2E0pt;
font-family:Arial'> &n=
bsp; =
; &n=
bsp; John
McCash<o:p></o:p></span></font></p>
</div>
<br>-----------------------------------------------------------------------=
-------------------------<br>This message is for the designated recipient o=
nly and may<br>contain privileged, proprietary, or otherwise private inform=
ation. <br>If you have received it in error, please notify the sender<br>i=
mmediately and delete the original. Any unauthorized use of<br>this email =
is prohibited.<br>---------------------------------------------------------=
---------------------------------------<br>[mf2]</body>
</html>
---- =_NextPart_ST_11_20_48_Friday_September_
10_2004_13802--
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
________________________________________
_______
Snort-users mailing list
Snort-users@lists.sourceforge.net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists...nfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf....ist=snort-users
| |
| sekure 2004-09-10, 5:46 pm |
| Well you can do something like:
select count(signature) as num, signature
from event
group by signature
order by num desc
limit 10;
This should give you the top 10 alerts in the event table. Keep in
mind though, that those may not necessarily be the ones that take up
the most disk space, since they may have a completely empty "data"
record associated with them.
Then you can look up the signature in the signature table.
Is this what you were looking for?
----- Original Message -----
From: McCash, John <john.mccash@andrew.com>
Date: Fri, 10 Sep 2004 11:20:47 -0500
Subject: [Snort-users] Finding alerts taking up the most database space
To: snort-users@lists.sourceforge.net
Hi,
I currently am running snort and acid with mysql, and
my database size is getting up around 6G. The data table, data.MYD
alone is about 3.3G. As you may imagine, my db performance is lousy.
Does anyone have an easy way of determining which alerts are taking up
the greatest amount of db space, so that I can selectively prune those
entries?
Thanks in advance
John McCash
------------------------------------------------------------------------------------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information.
If you have received it in error, please notify the sender
immediately and delete the original. Any unauthorized use of
this email is prohibited.
------------------------------------------------------------------------------------------------
[mf2]
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
________________________________________
_______
Snort-users mailing list
Snort-users@lists.sourceforge.net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists...nfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf....ist=snort-users
| |
| McCash, John 2004-09-10, 5:46 pm |
| Sekure,
No. I'm already doing that (through ACID). Unfortunately, the
alerts that are showing up in the greatest numbers appear to be only
about 20% of my problem. The real issue here appears to be that variable
amounts of packet capture are stored for many events, and there's no
index for the size of the packet capture record for a given event (or so
I believe...).
Thanks anyway
John
-----Original Message-----
From: sekure [mailto:sekure@gmail.com]=20
Sent: Friday, September 10, 2004 11:41 AM
To: McCash, John
Cc: snort-users@lists.sourceforge.net
Subject: Re: [Snort-users] Finding alerts taking up the most database
space
Well you can do something like:
select count(signature) as num, signature=20
from event=20
group by signature=20
order by num desc=20
limit 10;
This should give you the top 10 alerts in the event table. Keep in
mind though, that those may not necessarily be the ones that take up
the most disk space, since they may have a completely empty "data"
record associated with them.
Then you can look up the signature in the signature table.
Is this what you were looking for?
----- Original Message -----
From: McCash, John <john.mccash@andrew.com>
Date: Fri, 10 Sep 2004 11:20:47 -0500
Subject: [Snort-users] Finding alerts taking up the most database space
To: snort-users@lists.sourceforge.net
Hi,
I currently am running snort and acid with mysql, and
my database size is getting up around 6G. The data table, data.MYD
alone is about 3.3G. As you may imagine, my db performance is lousy.
Does anyone have an easy way of determining which alerts are taking up
the greatest amount of db space, so that I can selectively prune those
entries?
Thanks in advance
John McCash
------------------------------------------------------------------------
------------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information.=20
If you have received it in error, please notify the sender
immediately and delete the original. Any unauthorized use of
this email is prohibited.
------------------------------------------------------------------------
------------------------
[mf2]
---------------------------------------------------------------------------=
---------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information. =20
If you have received it in error, please notify the sender
immediately and delete the original. Any unauthorized use of
this email is prohibited.
---------------------------------------------------------------------------=
---------------------
[mf2]
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
________________________________________
_______
Snort-users mailing list
Snort-users@lists.sourceforge.net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists...nfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf....ist=snort-users
| |
| M Shirk 2004-09-13, 5:46 pm |
| When using an IDS in general, if you have 6G of data for a very short
time-frame, you may need to either tune your sensor by filtering, or by
archiving that data.
If this is for a business/project, you need to have a definition of the
time-frame to keep live data available for analysis. One of the clients I
worked with created 2 GB of data every 3 months. I knew what the problem
was, but they did not let us filter :-). They wanted this info for trending
(don't ask).
I think others on the list would chime in that this is not a snort problem
because snort is working.
Do you have snort and the mysql DB and your webserver all on the same
server? I have run this configuration just for testing and it kills my
rather old system with 160MB of RAM.
Shirkdog.
http://www.shirkdog.us
>From: "McCash, John" <John.McCash@andrew.com>
>To: <snort-users@lists.sourceforge.net>
>Subject: [Snort-users] Finding alerts taking up the most database space
>Date: Fri, 10 Sep 2004 11:20:47 -0500
>
>Hi,
>
> I currently am running snort and acid with mysql, and my
>database size is getting up around 6G. The data table, data.MYD alone is
>about 3.3G. As you may imagine, my db performance is lousy. Does anyone
>have an easy way of determining which alerts are taking up the greatest
>amount of db space, so that I can selectively prune those entries?
>
> Thanks in advance
>
> John McCash
>
>------------------------------------------------------------------------------------------------
>This message is for the designated recipient only and may
>contain privileged, proprietary, or otherwise private information.
>If you have received it in error, please notify the sender
>immediately and delete the original. Any unauthorized use of
>this email is prohibited.
>------------------------------------------------------------------------------------------------
>[mf2]
________________________________________
_________________________
Express yourself instantly with MSN Messenger! Download today - it's FREE!
hthttp://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
________________________________________
_______
Snort-users mailing list
Snort-users@lists.sourceforge.net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists...nfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf....ist=snort-users
| |
| McCash, John 2004-09-22, 10:25 pm |
| Hi Shirkdog,
The 6G was for a three month timeframe, which is my defined
retention period. I'm working on adding pass rules to filter the data,
and I've got my sensors, web server, and DB on different hosts. And the
DB runs on a fairly high performance server (even if it is on Windows
:-(). I didn't say it was a snort problem, I just asked if anyone knew
an easy way to pinpoint which alerts were taking up the most DB space.
You'd think that a 3.5GHz quad processor box with 4G of RAM, and a
(admittedly only 3 disk) raid array for the DB storage, could handle a
bit larger database size before choking... I've tuned the mysql
configuration as best I can given what I've been able to find on the
Internet, but there may be some database hacks that I've missed.
Thanks anyway
John
-----Original Message-----
From: snort-users-admin@lists.sourceforge.net
[mailto:snort-users-admin@lists.sourceforge.net] On Behalf Of M Shirk
Sent: Monday, September 13, 2004 11:32 AM
To: snort-users@lists.sourceforge.net
Subject: RE: [Snort-users] Finding alerts taking up the most database
space
When using an IDS in general, if you have 6G of data for a very short=20
time-frame, you may need to either tune your sensor by filtering, or by=20
archiving that data.
If this is for a business/project, you need to have a definition of the=20
time-frame to keep live data available for analysis. One of the clients
I=20
worked with created 2 GB of data every 3 months. I knew what the problem
was, but they did not let us filter :-). They wanted this info for
trending=20
(don't ask).
I think others on the list would chime in that this is not a snort
problem=20
because snort is working.
Do you have snort and the mysql DB and your webserver all on the same=20
server? I have run this configuration just for testing and it kills my=20
rather old system with 160MB of RAM.
Shirkdog.
http://www.shirkdog.us
>From: "McCash, John" <John.McCash@andrew.com>
>To: <snort-users@lists.sourceforge.net>
>Subject: [Snort-users] Finding alerts taking up the most database space
>Date: Fri, 10 Sep 2004 11:20:47 -0500
>
>Hi,
>
> I currently am running snort and acid with mysql, and
my
>database size is getting up around 6G. The data table, data.MYD alone
is
>about 3.3G. As you may imagine, my db performance is lousy. Does anyone
>have an easy way of determining which alerts are taking up the greatest
>amount of db space, so that I can selectively prune those entries?
>
> Thanks in advance
>
> John McCash
>
>-----------------------------------------------------------------------
-------------------------
>This message is for the designated recipient only and may
>contain privileged, proprietary, or otherwise private information.
>If you have received it in error, please notify the sender
>immediately and delete the original. Any unauthorized use of
>this email is prohibited.
>-----------------------------------------------------------------------
-------------------------
>[mf2]
________________________________________
_________________________
Express yourself instantly with MSN Messenger! Download today - it's
FREE!=20
hthttp://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.=20
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
________________________________________
_______
Snort-users mailing list
Snort-users@lists.sourceforge.net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists...nfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf....t=3Dsnort-users
---------------------------------------------------------------------------=
---------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information. =20
If you have received it in error, please notify the sender
immediately and delete the original. Any unauthorized use of
this email is prohibited.
---------------------------------------------------------------------------=
---------------------
[mf2]
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
________________________________________
_______
Snort-users mailing list
Snort-users@lists.sourceforge.net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists...nfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf....ist=snort-users
|
|
|
|
|