[Openais] [Cluster-devel] cluster/logging settings

[Steven Dake]

On Fri, 2008-10-31 at 06:28 +0100, Fabio M. Di Nitto wrote:
> On Thu, 30 Oct 2008, David Teigland wrote:
> 
> >> If I set log_level to LOG_INFO, I do expect to see both critblabla and
> >> infoblabla on all selected outputs.
> >
> > Note that you've said "log_level", but it's actually "syslog_level", which
> > might lead people to different conclusions about what the option does.
> 
> I meant syslog_level. You are right.
> 
> void logt_print(int level, char *fmt, ...)
> {
>          va_list ap;
> 
>          if (level > logt_priority)
>                  return;
> 
> we send a logt_print(LOG_DEBUG..
> but if syslog_level is set to LOG_INFO, that message will not make it 
> anywhere, even if debug=on.
> 
> >
> >> If I set log_level to LOG_DEBUG, I clearly expect to see debugblabla
> >> passing throgh as well.
> >> (except if filtering to syslog is enabled, but we already agreed on this
> >> as required feature so I won't mention it as special case anylonger).
> >>
> >> I often expect that enabling LOG_DEBUG as priority, it will also enable
> >> debugging code in general and viceversa. If I set debug=on, I want to be
> >> able to catch LOG_DEBUG automatically, because i assume that most of the
> >> LOG_DEBUG is wrapped between if(debug) { } statements. I don't feel the
> >> need to set two options to enable full debugging.
> >
> > You don't need to.  Under my scheme, you set debug=on, and full debugging
> > appears in the log file where most people want it, and it doesn't appear
> > in syslog where most people wouldn't want it.
> 
> Not really.. see below the code snippet...
> 
> >  If someone *does* want all
> > debugging to appear in syslog, then they set debug=on and
> > syslog_level=debug to get it... (and once they see the result, they'll
> > change it back, because it's really not nice to see all that in
> > /var/log/messages.)
> 
> We all agreed on this already that's why we have the 
> LOG_MODE_FILTER_DEBUG_FROM_SYSLOG that we should probably make a config 
> option in cluster.conf or simply avoid by default.
> 
> > Under your scheme, syslog_level and debug change each other in confusing
> > and redundant ways.
> 
> It's not really confusing.. debug enables LOG_DEBUG and LOG_DEBUG enables 
> debug. The only thing that might be confusing is that you need to take 
> into account global debug, (sub)system debug setting and priority.
> 
> >  By setting debug=on you automatically get all
> > debugging in syslog (in addition to the logfile usually), which is not
> > where we wanted it...  So we added yet another complication:
> > LOG_MODE_FILTER_DEBUG_FROM_SYSLOG, which counteracts the bad effects of
> > the debug/syslog_level interaction which we didn't really want in the
> > first place.  See how complicated this gets when you have one option
> > changing another one, but not quite the way you want, so you add another
> > flag to work around the unintended effects of one setting implicitly
> > changing another?  It's bad all around: just keep them independent and all
> > the pain goes away.
> >
> > (We wouldn't need FILTER_DEBUG_FROM_SYSLOG in my scheme because you
> > control each source and destination point explicitly; you say exactly what
> > you want, and get it.)
> 
> Unless I am misreading your code:
> 
> void logt_print(int level, char *fmt, ...)
> {
>          va_list ap;
> 
>          if (level > logt_priority)
>                  return;
> 
> we send a logt_print(LOG_DEBUG..
> if syslog_level is set to LOG_INFO, that message will not make it
> anywhere, even if debug=on.
> 
> maybe debug=on should really set syslog_level=LOG_DEBUG and 
> LOG_MODE_FILTER_DEBUG_FROM_SYSLOG to make this really happens
> 

Here is my take on how logging should work and does work in logsysv2.

There are two types of messages.  Those intended for users/admins and
those intended for developers.

Both of these message types should always be recorded *somewhere*.  The
entire concept of "LOG_LEVEL_DEBUG" is dubious to me.  If you want to
stick with that symanetic and definition that is fine, but really a
LOG_LEVEL_DEBUG means "this message is for the developer".  These
messages should be recorded and stored when a process segfaults, aborts
due to assertion, or at administrative request.  Since the frequency of
these messages is high there is no other option for recording them since
they must _always_ be recorded for the purposes of debugging a field
failure.  Recording to disk or syslog has significant performance
impact.

The only solution for these types of messages is to record them into a
flight recorder buffer which can be dumped:
1) at segv
2) at sigabt
3) at administrative request

This is a fundamental difference in how we have approached logging
debugging messages in the past but will lead to the ability to ensure we
_always_ have debug trace data available instead of telling the
user/admin "Go turn on debug and hope you can reproduce that error and
btw since 100000k messages are logged your disk will fill up with
irrelevant debug messages and your system will perform like mud".

Logging these in memory is the only solution that I see as suitable and
in all cases they should be filtered from any output source such as
stderr, file, or syslog.

Regards
-steve


 
> Fabio
> 
> --
> I'm going to make him an offer he can't refuse.
>