- The shared files Windows implementation introduced in PR #3132 works
in multi-process single-threaded contexts but it doesn't work
correctly in single-process multi-threaded contexts.
- The issue is that the LockFileEx Win32 function works on a per-handle
basis.
- In a multi-process context, each process will have called
SharedFiles::add_new_handler when initializing the SharedFile and
obtained a handle, and thus locking will work.
- When running ModSecurity in a single process using multiple threads,
the initialization of the SharedFile will happen once and the handle
will be shared by all threads. Then, if two threads try to write to
the same shared file concurrently, they may deadlock as one of them
will lock the file (by calling LockFileEx) and then proceed to write
to the file. If before writing to the file and unlocking it, another
thread calls LockFileEx on the same handle, the attempt to write to
the file will lock generating a deadlock.
- The new implementation replaces usage of LockFileEx/UnlockFileEx with
a named mutex to lock access to the shared file.
- A named mutex is used to support multi-process scenarios.
- The mutex name is generated using the filename to support multiple
shared files (such as that for the debug and audit logs).
- This assumes that both process will initialize the SharedFile
instance using the same filename (which is expected as they'd be
using the same configuration file)
- In Windows build, replaced usage of fcntl with cmd F_SETLKW with
Win32 APIs to do file locking (LockFileEx & UnlockFileEx).
- Reintroduced the reference counting initially present in the class
which is necessary to correctly handle merging of rules. This allows
for correctly closing the file and removing the associated entry from
m_handlers when the file is no longer used.
- The need for reference counting can be seen in the example
simple_example_using_c, where rules are initially loaded locally and
then further rules are loaded remotely. This will initially open a
shared file for a log, then in order to merge rules, the shared file
is opened again for the new configuration. Then, the previous
configuration closes the shared file on destruction. That is, two
consecutive opens are done on a shared file, which is followed by
a close. If the shared file is not reference counted, the shared file
will be closed while there is still a reference active. The current
version works because closing of the file has been disabled after
reference counting was removed.
- Replaced `std::vector` data structure with `std::unordered_map` to
improve lookup/update times, and simplify code.
- Removed unused code
- Shared memory to store msc_file_handler structure
- Initially SharedFiles used shared memory to store information
about each shared file, including its file pointer and a mutex to
synchronize access to the file on write. See code at commit 01c13da,
in particular, usage of lock & fp fields in the msc_file_handler_t
structure.
- At that time, msc_file_handler_t included reference counting too
with the using_it field, which was incremented when a file was
opened and decremented on close. If the reference count reached
zero, the shared file would be closed, the lock destroyed and the
file handler entry removed from m_handlers.
- Reference counting was removed in commit 7f9cd76, which
introduced the following issues in SharedFiles::close:
- No longer closes the file pointer.
- The file pointer appears to be reset when a.second = 0, but
this is a local copy of the data pair obtained from m_handlers,
so this is essentially a nop (updating a local variable that is
not referenced later in the function).
- NOTE: The file pointer was moved out of the shared memory in
this commit too, and stored alongside the msc_file_handler_t
instance in the m_handlers entry associated to the shared file.
- The lock is no longer destroyed.
- The shared memory is marked to be destroyed in the call to:
shmctl(a.first->shm_id_structure, IPC_RMID, NULL);
- The shared file entry is not removed from m_handlers, so:
- the file pointer is still valid, which is how writing to the
file continues to work,
- the reference to the shared memory is also present and will
be marked to be destroyed whenever close is called again on the
shared file.
- File locking using the mutex in msc_file_handler_t was replaced in
commit 3d20304 with usage of fcntl with cmd F_SETLKW.
- At this time, it appears that the shared memory is no longer used,
as the file pointer and locking no longer depend on it.
- MODSEC_USE_GENERAL_LOCK
- This code is introduced commit 7f9cd76 and is enabled if
MODSEC_USE_GENERAL_LOCK` is defined.
- The define is commented out in the source code since the original
commit and is not present in the build configuration either.
- In commit ff9152e, in the SharedFiles constructor, the
initialization of the local variable toBeCreated is removed. This
means that in this version, if MODSEC_USE_GENERAL_LOCK is enabled,
execution of the code that checks on toBeCreated is undefined.
- Then, in commit 9b40a04, the variable toBeCreated is initialized
again, but is now set to false, which means that if
MODSEC_USE_GENERAL_LOCK is enabled, the shared memory and lock it
uses will *not* be initialized and thus doesn't work (execution of
the current version will result in trying to acquire a lock that
will be null).
- I conclude that the feature is not used and can be removed.
- Additionally, if it were working, I think the lock should be
used in SharedFiles::write as well, which is a reader of the
underlying data structures protected by this lock when they're
modified in SharedFiles::open & SharedFiles::close.
When reloading Nginx, there is a race condition which is visible under high
load. As the logging mutex is shared between multiple workers, when a worker
is sent a stop signal during a reload, and the log mutex is held, write()
will never return, which means that the mutex will never unlock. As other
workers share this mutex, they will deadlock.
fcntl does not suffer from this issue.
