[OE-core] [pseudo][PATCH] Fix to fcntl guts to ignore flags that can be ORed into cmd

[Will Page]

The fcntl guts switch on "cmd" parameter to identify the fcntl
command being issued, but isn't aware of the file creation flags that
can be ORed in.

This change masks out the flags from the command only for the switch
statement so that it can correctly determine whether it needs to pass
"lock" or "arg".  When real_fcntl() is called, the original value is
still passed on. This corrects an issue observed using pseudo on modern
linux desktops resulting in "pseudo: unknown fcntl argument 1030,
assuming long argument." diagnostics in the output.  Decimal 1030 is
0x0406, which translates to O_NOCTTY | F_SETLK, and should call
"rc = real_fcntl(fd, cmd, lock);", but instead falls through to the
default case instead calling "rc = real_fcntl(fd, cmd, arg);".

Tested the change using perftest - without this patch, the issue is
trivially reproducible on my Ubuntu Xenial system, and after patching it
emitted no "unknown fcntl argument" diagnostics.

Signed-off-by: Will Page <Will.Page at ni.com>
---
 ports/linux/guts/fcntl.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/ports/linux/guts/fcntl.c b/ports/linux/guts/fcntl.c
index 639fd24..d278a8c 100644
--- a/ports/linux/guts/fcntl.c
+++ b/ports/linux/guts/fcntl.c
@@ -8,6 +8,10 @@
  */
  	long arg;
 	int save_errno;
+	/* some bits can be ORed into the cmd should be explicitly ignored
+	 * see fcntl documentation on F_SETFL */
+	int o_mode_mask = (O_RDONLY | O_WRONLY | O_RDWR) |
+		(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC);
 
 	/* we don't know whether we need lock or arg; grab both, which
 	 * should be safe enough on Linuxy systems. */
@@ -15,7 +19,7 @@
 	arg = va_arg(ap, long);
 	va_end(ap);
 
-	switch (cmd) {
+	switch (cmd & ~(o_mode_mask)) {
 	case F_DUPFD:
 #ifdef F_DUPFD_CLOEXEC
 	case F_DUPFD_CLOEXEC:
-- 
2.7.4