- The process config parameters 'stdout_capturefile' and

    'stderr_capturefile' are no longer valid.  They have been replaced
    with the 'stdout_capture_maxbytes' and 'stderr_capture_maxbytes'
    parameters, which are meant to be suffix-multipled integers.  They
    both default to zero.  When they are zero, process communication
    event capturing is not performed.  When either is nonzero, the
    value represents the maximum number of bytes that will be captured
    between process event start and end tags.  This change was to
    support the fact that we no longer keep capture data in a separate
    file, we just use a FIFO in RAM to maintain capture info.  For
    users whom don't care about process communication events, or whom
    haven't changed the defaults for 'stdout_capturefile' or
    'stderr_capturefile', they needn't do anything to their
    configurations to deal with this change.

CHANGES.txt

@@ -65,6 +65,21 @@ Next Release
 
   - Improve process communication event performance.
 
+  - The process config parameters 'stdout_capturefile' and
+    'stderr_capturefile' are no longer valid.  They have been replaced
+    with the 'stdout_capture_maxbytes' and 'stderr_capture_maxbytes'
+    parameters, which are meant to be suffix-multipled integers.  They
+    both default to zero.  When they are zero, process communication
+    event capturing is not performed.  When either is nonzero, the
+    value represents the maximum number of bytes that will be captured
+    between process event start and end tags.  This change was to
+    support the fact that we no longer keep capture data in a separate
+    file, we just use a FIFO in RAM to maintain capture info.  For
+    users whom don't care about process communication events, or whom
+    haven't changed the defaults for 'stdout_capturefile' or
+    'stderr_capturefile', they needn't do anything to their
+    configurations to deal with this change.
+
 3.0a2
 
   - Fixed the README.txt example for defining the supervisor RPC

README.txt

@@ -394,11 +394,11 @@ Configuration File '[program:x]' Section Settings
     stdout_logfile=AUTO
     stdout_logfile_maxbytes=50MB
     stdout_logfile_backups=10
-    stdout_capturefile=NONE
+    stdout_capture_maxbytes=1MB
     stderr_logfile=AUTO
     stderr_logfile_maxbytes=50MB
     stderr_logfile_backups=10
-    stderr_capturefile=NONE
+    stderr_capture_maxbytes=1MB
     environment=A=1,B=2
 
   '[program:foo]' -- the section header, required for each program.
@@ -514,13 +514,12 @@ Configuration File '[program:x]' Section Settings
   this to 0 to indicate an unlimited number of backups.  Default: 10.
   (New in 3.0, replaces "logfile_backups")
 
-  'stdout_capturefile' -- file written to when process is in "stdout
-  capture mode" (see "Capture Mode and Process Communication Events"
-  later in this document).  May be a file path, NONE, or AUTO.  The
-  stdout_capturefile value can contain Python string expressions that
-  will evaluated against a dictionary that contains the keys
-  "process_num", "program_name" and "group_name".  Default: NONE.
-  (New in 3.0)
+  'stdout_capture_maxbytes' -- max number of bytes written to FIFO
+  when process is in "stdout capture mode" (see "Capture Mode and
+  Process Communication Events" later in this document).  Should be an
+  integer (suffix multipliers like "KB", "MB" and "GB" can used in the
+  value).  If this value is 0, process capture mode will be off.
+  Default: 0.  (New in 3.0)
 
   'stderr_logfile' -- Put process stderr output in this file unless
   redirect_stderr is true.  Accepts the same value types as
@@ -535,11 +534,12 @@ Configuration File '[program:x]' Section Settings
   resulting from process stderr log file rotation.  Default: 10.  (New
   in 3.0)
 
-  'stderr_capturefile' -- file written to when process is in "stderr
-  capture mode" (see "Capture Mode and Process Communication Events"
-  later in this document).  May contain the same Python string
-  expressions as "stdout_capturefile". May be a file path, NONE, or
-  AUTO.  Default: NONE.  (New in 3.0)
+  'stderr_capture_maxbytes' -- max number of bytes written to FIFO
+  when process is in "stderr capture mode" (see "Capture Mode and
+  Process Communication Events" later in this document).  Should be an
+  integer (suffix multipliers like "KB", "MB" and "GB" can used in the
+  value).  If this value is 0, process capture mode will be off.
+  Default: 0.  (New in 3.0)
 
   'environment' -- A list of key/value pairs in the form
   "KEY=val,KEY2=val2" that will be placed in the child process'
@@ -633,10 +633,10 @@ Configuration File '[eventlistener:x]' Section Settings (New in 3.0)
     environment=A=1,B=2
 
   Note that all the options available to '[program:x]' sections are
-  respected by eventlistener sections except for "stdout_capturefile"
-  and "stderr_capturefile" (event listeners cannot emit process
-  communication events, see "Capture Mode and Process Communication
-  Events" elsewhere in this document).
+  respected by eventlistener sections except for
+  "stdout_capture_maxbytes" and "stderr_capture_maxbytes" (event
+  listeners cannot emit process communication events, see "Capture
+  Mode and Process Communication Events" elsewhere in this document).
 
   '[eventlistener:x]' sections have two keys which '[program:x]'
   sections do not have:
@@ -1257,26 +1257,31 @@ Event Listener Error Conditions
 Capture Mode and Process Communication Events (New in 3.0)
 
   If a '[program:x]' section in the configuration file defines a
-  "stdout_capturefile" or "stderr_capturefile" parameter, each process
-  represented by the program section may emit special tokens on its
-  stdout or stderr stream (respectively) which will effectively cause
-  supervisor to emit a "PROCESS_COMMUNICATION" event type.
+  non-zero "stdout_capture_maxbytes" or "stderr_capture_maxbytes"
+  parameter, each process represented by the program section may emit
+  special tokens on its stdout or stderr stream (respectively) which
+  will effectively cause supervisor to emit a "PROCESS_COMMUNICATION"
+  event type.
 
   The process communications protocol relies on two tags, one which
   commands supervisor to enter "capture mode" for the stream and one
   which commands it to exit.  When a process stream enters "capture
-  mode", data sent to the stream will be sent to a separate logfile
-  (the "capturefile").  When a process stream exits capture mode, the
-  data in the capturefile is read into memory (a maximum of 2MB), and
-  a PROCESS_COMMUNICATION event is emitted by supervisor, which may be
-  intercepted by event listeners.
+  mode", data sent to the stream will be sent to a separate buffer in
+  memory, the "capture buffer", which is allowed to contain a maximum
+  of 'capture_maxbytes'.  During capture mode, when the buffer's
+  length exceeds 'capture_maxbytes' bytes, the earliest data in the
+  buffer is discarded to make room for new data.  When a process
+  stream exits capture mode, the data in the capture buffer is read
+  into memory, and a PROCESS_COMMUNICATION event is emitted by
+  supervisor, which may be intercepted by event listeners.
 
   The tag to begin "capture mode" in a process stream is
   '<!--XSUPERVISOR:BEGIN-->'.  The tag to exit capture mode is
   '<!--XSUPERVISOR:END-->'.  The data between these tags may be
   arbitrary, and forms the payload of the PROCESS_COMMUNICATION event.
-  For example, if a program is set up with a stdout_capturefile, and
-  it emits the following on its stdout stream::
+  For example, if a program is set up with (e.g.) a
+  stdout_capture_maxbytes=1MB, and it emits the following on its
+  stdout stream::
 
     <!--XSUPERVISOR:BEGIN-->Hello!<!--XSUPERVISOR:END-->
 

TODO.txt

@@ -1,6 +1,3 @@
-- Get rid of stdout|err_capturelog process config option in favor of a
-  boolean.
-
 - Add a new cwd option that will chdir after the fork-exec.
 
 - Move set of 'SUPERVISOR_ENABLED' envvar to process spawn method.

sample.conf

@@ -9,7 +9,7 @@ http_port=/tmp/supervisor.sock ; (default is to run a UNIX domain socket server)
 logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
 logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
 logfile_backups=10          ; (num of main logfile rotation backups;default 10)
-loglevel=info               ; (logging level;default info; others: debug,warn)
+loglevel=info               ; (log level;default info; others: debug,warn,trace)
 pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
 nodaemon=false              ; (start in foreground if true;default false)
 minfds=1024                 ; (min. avail startup file descriptors;default 1024)
@@ -56,11 +56,11 @@ serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
 ;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
 ;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
 ;stdout_logfile_backups=10     ; # of stdout logfile backups (default 10)
-;stdout_capturefile=NONE       ; stdout capturemode data file (default AUTO)
+;stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
 ;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
 ;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
 ;stderr_logfile_backups=10     ; # of stderr logfile backups (default 10)
-;stderr_capturefile=NONE       ; stderr capturemode data file (default AUTO)
+;stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
 ;environment=A=1,B=2           ; process environment additions
 
 ; The below sample eventlistener section shows all possible

src/supervisor/dispatchers.py

@@ -68,7 +68,6 @@ class POutputDispatcher(PDispatcher):
     capturemode = False # are we capturing process event data
     mainlog = None #  the process' "normal" logger
     capturelog = None # the logger while we're in capturemode
-    capturefile = None # the capture file name
     childlog = None # the current logger (event or main)
     output_buffer = '' # data waiting to be logged
 
@@ -79,7 +78,8 @@ class POutputDispatcher(PDispatcher):
         self.channel = channel = self.event_type.channel
 
         logfile = getattr(process.config, '%s_logfile' % channel)
-        capturefile = getattr(process.config, '%s_capturefile' % channel)
+        capture_maxbytes = getattr(process.config,
+                                   '%s_capture_maxbytes' % channel)
 
         if logfile:
             maxbytes = getattr(process.config, '%s_logfile_maxbytes' % channel)
@@ -92,14 +92,13 @@ class POutputDispatcher(PDispatcher):
                 maxbytes=maxbytes,
                 backups=backups)
 
-        if capturefile:
-            self.capturefile = capturefile
+        if capture_maxbytes:
             self.capturelog = self.process.config.options.getLogger(
                 None, # BoundIO
                 loggers.LevelsByName.INFO,
                 '%(message)s',
                 rotating=False,
-                maxbytes=1 << 21, #2MB
+                maxbytes=capture_maxbytes,
                 )
 
         self.childlog = self.mainlog
@@ -177,7 +176,6 @@ class POutputDispatcher(PDispatcher):
             if self.capturemode:
                 self.childlog = self.capturelog
             else:
-                capturefile = self.capturefile
                 for handler in self.capturelog.handlers:
                     handler.flush()
                 data = self.capturelog.getvalue()

src/supervisor/options.py

@@ -633,6 +633,8 @@ class ServerOptions(Options):
         numprocs = integer(get(section, 'numprocs', 1))
         process_name = get(section, 'process_name', '%(program_name)s')
         environment_str = get(section, 'environment', '')
+        stdout_cmaxbytes = byte_size(get(section,'stdout_capture_maxbytes','0'))
+        stderr_cmaxbytes = byte_size(get(section,'stderr_capture_maxbytes','0'))
 
         command = get(section, 'command', None)
         if command is None:
@@ -648,6 +650,7 @@ class ServerOptions(Options):
                     'numprocs > 1')
 
         for n in ('stdout_logfile', 'stderr_logfile'):
+            # do warning
             lf_name = logfile_name(get(section, n, Automatic))
             mb_key = '%s_maxbytes' % n
             maxbytes = byte_size(get(section, mb_key, '50MB'))
@@ -675,12 +678,6 @@ class ServerOptions(Options):
                     val = expand(val, expansions, n)
                 logfiles[n] = val
 
-                n = '%s_capturefile' % k
-                val = logfile_name(get(section, n, None))
-                if isinstance(val, basestring):
-                    val = expand(val, expansions, n)
-                logfiles[n] = val
-
                 bu_key = '%s_logfile_backups' % k
                 backups = integer(get(section, bu_key, 10))
                 logfiles[bu_key] = backups
@@ -700,11 +697,11 @@ class ServerOptions(Options):
                 startretries=startretries,
                 uid=uid,
                 stdout_logfile=logfiles['stdout_logfile'],
-                stdout_capturefile=logfiles['stdout_capturefile'],
+                stdout_capture_maxbytes = stdout_cmaxbytes,
                 stdout_logfile_backups=logfiles['stdout_logfile_backups'],
                 stdout_logfile_maxbytes=logfiles['stdout_logfile_maxbytes'],
                 stderr_logfile=logfiles['stderr_logfile'],
-                stderr_capturefile=logfiles['stderr_capturefile'],
+                stderr_capture_maxbytes = stderr_cmaxbytes,
                 stderr_logfile_backups=logfiles['stderr_logfile_backups'],
                 stderr_logfile_maxbytes=logfiles['stderr_logfile_maxbytes'],
                 stopsignal=stopsignal,
@@ -1300,9 +1297,9 @@ class Config:
 class ProcessConfig(Config):
     def __init__(self, options, name, command, priority, autostart,
                  autorestart, startsecs, startretries, uid,
-                 stdout_logfile, stdout_capturefile,
+                 stdout_logfile, stdout_capture_maxbytes,
                  stdout_logfile_backups, stdout_logfile_maxbytes,
-                 stderr_logfile, stderr_capturefile,
+                 stderr_logfile, stderr_capture_maxbytes,
                  stderr_logfile_backups, stderr_logfile_maxbytes,
                  stopsignal, stopwaitsecs, exitcodes, redirect_stderr,
                  environment=None):
@@ -1316,11 +1313,11 @@ class ProcessConfig(Config):
         self.startretries = startretries
         self.uid = uid
         self.stdout_logfile = stdout_logfile
-        self.stdout_capturefile = stdout_capturefile
+        self.stdout_capture_maxbytes = stdout_capture_maxbytes
         self.stdout_logfile_backups = stdout_logfile_backups
         self.stdout_logfile_maxbytes = stdout_logfile_maxbytes
         self.stderr_logfile = stderr_logfile
-        self.stderr_capturefile = stderr_capturefile
+        self.stderr_capture_maxbytes = stderr_capture_maxbytes
         self.stderr_logfile_backups = stderr_logfile_backups
         self.stderr_logfile_maxbytes = stderr_logfile_maxbytes
         self.stopsignal = stopsignal
@@ -1338,10 +1335,6 @@ class ProcessConfig(Config):
             self.stdout_logfile = get_autoname(name, sid, 'stdout')
         if self.stderr_logfile is Automatic:
             self.stderr_logfile = get_autoname(name, sid, 'stderr')
-        if self.stdout_capturefile is Automatic:
-            self.stdout_capturefile = get_autoname(name, sid, 'stdout_capture')
-        if self.stderr_capturefile is Automatic:
-            self.stderr_capturefile = get_autoname(name, sid, 'stderr_capture')
             
     def make_process(self, group=None):
         from supervisor.process import Subprocess

src/supervisor/tests/base.py

@@ -399,9 +399,9 @@ class DummyProcess:
 class DummyPConfig:
     def __init__(self, options, name, command, priority=999, autostart=True,
                  autorestart=True, startsecs=10, startretries=999,
-                 uid=None, stdout_logfile=None, stdout_capturefile=None,
+                 uid=None, stdout_logfile=None, stdout_capture_maxbytes=0,
                  stdout_logfile_backups=0, stdout_logfile_maxbytes=0,
-                 stderr_logfile=None, stderr_capturefile=None,
+                 stderr_logfile=None, stderr_capture_maxbytes=0,
                  stderr_logfile_backups=0, stderr_logfile_maxbytes=0,
                  redirect_stderr=False,
                  stopsignal=None, stopwaitsecs=10,
@@ -416,11 +416,11 @@ class DummyPConfig:
         self.startretries = startretries
         self.uid = uid
         self.stdout_logfile = stdout_logfile
-        self.stdout_capturefile = stdout_capturefile
+        self.stdout_capture_maxbytes = stdout_capture_maxbytes
         self.stdout_logfile_backups = stdout_logfile_backups
         self.stdout_logfile_maxbytes = stdout_logfile_maxbytes
         self.stderr_logfile = stderr_logfile
-        self.stderr_capturefile = stderr_capturefile
+        self.stderr_capture_maxbytes = stderr_capture_maxbytes
         self.stderr_logfile_backups = stderr_logfile_backups
         self.stderr_logfile_maxbytes = stderr_logfile_maxbytes
         self.redirect_stderr = redirect_stderr

src/supervisor/tests/test_dispatchers.py

@@ -60,7 +60,7 @@ class POutputDispatcherTests(unittest.TestCase):
         options = DummyOptions()
         options.readfd_result = 'abc'
         config = DummyPConfig(options, 'process1', '/bin/process1',
-                              stdout_capturefile='abc')
+                              stdout_capture_maxbytes=100)
         process = DummyProcess(config)
         dispatcher = self._makeOne(process)
         self.assertEqual(dispatcher.handle_read_event(), None)
@@ -85,7 +85,7 @@ class POutputDispatcherTests(unittest.TestCase):
         from StringIO import StringIO
         config = DummyPConfig(options, 'process1', '/bin/process1',
                               stdout_logfile='/tmp/foo',
-                              stdout_capturefile='/tmp/capture')
+                              stdout_capture_maxbytes=500)
         process = DummyProcess(config)
         process.pid = 4000
         dispatcher = self._makeOne(process)
@@ -147,7 +147,7 @@ class POutputDispatcherTests(unittest.TestCase):
         options = DummyOptions()
         config = DummyPConfig(options, 'process1', '/bin/process1',
                               stdout_logfile='/tmp/foo',
-                              stdout_capturefile='/tmp/capture')
+                              stdout_capture_maxbytes=100)
         process = DummyProcess(config)
         dispatcher = self._makeOne(process)
         dispatcher.output_buffer = 'stdout string longer than a token'
@@ -164,7 +164,7 @@ class POutputDispatcherTests(unittest.TestCase):
         options = DummyOptions()
         config = DummyPConfig(options, 'process1', '/bin/process1',
                               stdout_logfile='/tmp/foo',
-                              stdout_capturefile='/tmp/capture')
+                              stdout_capture_maxbytes=100)
         process = DummyProcess(config)
         dispatcher = self._makeOne(process)
         dispatcher.output_buffer = 'a'
@@ -188,12 +188,9 @@ class POutputDispatcherTests(unittest.TestCase):
         from supervisor.loggers import getLogger
         options.getLogger = getLogger # actually use real logger
         logfile = '/tmp/log'
-        capturefile = '/tmp/capture'
         config = DummyPConfig(options, 'process1', '/bin/process1',
                               stdout_logfile=logfile,
-                              stdout_capturefile=capturefile)
-        config.stdout_logfile = logfile
-        config.capturefile = capturefile
+                              stdout_capture_maxbytes=1000)
         process = DummyProcess(config)
         dispatcher = self._makeOne(process)
 
@@ -216,10 +213,6 @@ class POutputDispatcherTests(unittest.TestCase):
                 os.remove(logfile)
             except (OSError, IOError):
                 pass
-            try:
-                os.remove(capturefile)
-            except (OSError, IOError):
-                pass
         
     def test_stdout_capturemode_multiple_buffers(self):
         from supervisor.events import ProcessCommunicationEvent
@@ -245,12 +238,9 @@ class POutputDispatcherTests(unittest.TestCase):
         from supervisor.loggers import getLogger
         options.getLogger = getLogger # actually use real logger
         logfile = '/tmp/log'
-        capturefile = '/tmp/capture'
         config = DummyPConfig(options, 'process1', '/bin/process1',
                               stdout_logfile=logfile,
-                              stdout_capturefile=capturefile)
-        config.stdout_logfile = logfile
-        config.capturefile = capturefile
+                              stdout_capture_maxbytes=10000)
         process = DummyProcess(config)
         dispatcher = self._makeOne(process)
         try:
@@ -286,10 +276,6 @@ class POutputDispatcherTests(unittest.TestCase):
                 os.remove(logfile)
             except (OSError, IOError):
                 pass
-            try:
-                os.remove(capturefile)
-            except (OSError, IOError):
-                pass
 
     def test_strip_ansi(self):
         options = DummyOptions()
@@ -321,7 +307,6 @@ class POutputDispatcherTests(unittest.TestCase):
         self.assertEqual(dispatcher.process, process)
         self.assertEqual(dispatcher.channel, 'stdout')
         self.assertEqual(dispatcher.fd, 0)
-        self.assertEqual(dispatcher.capturefile, None)
         self.assertEqual(dispatcher.capturelog, None)
         self.assertEqual(dispatcher.mainlog, None)
         self.assertEqual(dispatcher.childlog, None)
@@ -335,7 +320,6 @@ class POutputDispatcherTests(unittest.TestCase):
         self.assertEqual(dispatcher.process, process)
         self.assertEqual(dispatcher.channel, 'stdout')
         self.assertEqual(dispatcher.fd, 0)
-        self.assertEqual(dispatcher.capturefile, None)
         self.assertEqual(dispatcher.capturelog, None)
         self.assertEqual(dispatcher.mainlog.__class__, DummyLogger)
         self.assertEqual(dispatcher.childlog, dispatcher.mainlog)
@@ -343,13 +327,12 @@ class POutputDispatcherTests(unittest.TestCase):
     def test_ctor_capturelog_only(self):
         options = DummyOptions()
         config = DummyPConfig(options, 'process1', '/bin/process1',
-                              stdout_capturefile='/tmp/foo')
+                              stdout_capture_maxbytes=300)
         process = DummyProcess(config)
         dispatcher = self._makeOne(process)
         self.assertEqual(dispatcher.process, process)
         self.assertEqual(dispatcher.channel, 'stdout')
         self.assertEqual(dispatcher.fd, 0)
-        self.assertEqual(dispatcher.capturefile, '/tmp/foo')
         self.assertEqual(dispatcher.capturelog.__class__,DummyLogger)
         self.assertEqual(dispatcher.mainlog, None)
         self.assertEqual(dispatcher.childlog, None)
@@ -362,7 +345,6 @@ class POutputDispatcherTests(unittest.TestCase):
         self.assertEqual(dispatcher.process, process)
         self.assertEqual(dispatcher.channel, 'stdout')
         self.assertEqual(dispatcher.fd, 0)
-        self.assertEqual(dispatcher.capturefile, None)
         self.assertEqual(dispatcher.capturelog, None)
         self.assertEqual(dispatcher.mainlog, None)
         self.assertEqual(dispatcher.childlog, None)

src/supervisor/tests/test_options.py

@@ -366,7 +366,7 @@ class ServerOptionsTests(unittest.TestCase):
         self.assertEqual(pconfig.startretries, 100)
         self.assertEqual(pconfig.uid, 0)
         self.assertEqual(pconfig.stdout_logfile, None)
-        self.assertEqual(pconfig.stdout_capturefile, None)
+        self.assertEqual(pconfig.stdout_capture_maxbytes, 0)
         self.assertEqual(pconfig.stdout_logfile_maxbytes, 104857600)
         self.assertEqual(pconfig.stopsignal, signal.SIGKILL)
         self.assertEqual(pconfig.stopwaitsecs, 100)
@@ -685,9 +685,9 @@ class TestProcessConfig(unittest.TestCase):
         defaults = {}
         for name in ('name', 'command', 'priority', 'autostart', 'autorestart',
                      'startsecs', 'startretries', 'uid',
-                     'stdout_logfile', 'stdout_capturefile',
+                     'stdout_logfile', 'stdout_capture_maxbytes',
                      'stdout_logfile_backups', 'stdout_logfile_maxbytes',
-                     'stderr_logfile', 'stderr_capturefile',
+                     'stderr_logfile', 'stderr_capture_maxbytes',
                      'stderr_logfile_backups', 'stderr_logfile_maxbytes',
                      'stopsignal', 'stopwaitsecs', 'exitcodes',
                      'redirect_stderr', 'environment'):