(expand-file-name "/..")

[Marcus Harnisch]

Evaluating the expression below leads to strange results that are
contradicting the doc-string which claims that "." and ".." would be
removed.

(expand-file-name "/..")
=> "/.."

The good news is that at least this is FSF compatible behavior. FSF
Emacs documents this weirdness saying:

“For technical reasons, this function can return correct but
non-intuitive results for the root directory; for instance,
(expand-file-name ".." "/") returns "/..". For this reason, use
(directory-file-name (file-name-directory dirname)) to traverse a
filesystem tree, not (expand-file-name ".." dirname).”

Their suggested workaround is of course not correct in cases where
`dirname' ends with `directory-sep-char':

(directory-file-name (file-name-directory "/foo/bar/../"))
=> "/foo/bar/.." <<<< WRONG

(expand-file-name "/foo/bar/../")
=> "/foo/" <<<< RIGHT

It doesn't occur to me why it would be difficult to return

(expand-file-name "/..")
=> "/"

and

(expand-file-name "/.")
=> "/"

respectively. History?

Looking at the code I find this comment:
/* `/../' is the "superroot" on certain file systems. */

Is there any file system still in use with this concept?

Thanks
Marcus

[Stephen J. Turnbull]

Thanks for the report and investigating GNU Emacs, Marcus!

Post by Marcus Harnisch
Evaluating the expression below leads to strange results that are
contradicting the doc-string which claims that "." and ".." would be
removed.
(expand-file-name "/..")
=> "/.."

I suppose the problem is that this expression doesn't have
well-defined semantics since conceptually it takes you out of the file
system. Unix file systems do define this concept since all
directories have a parent pointer (AFAIK), and all set /.. to /
(AFAIK). But what this might mean in a filesystem without parent
pointers (DOS at least, also Windows NT, VMS IIRC) is unclear. On
those systems "/" is not an absolute path! (You need a drive letter.)

Note that this is a textual transformation; AFAIK the only system call
made in the function is to read /etc/passwd to expand ~USER.

Post by Marcus Harnisch
The good news is that at least this is FSF compatible behavior.

I think this is a killer argument for documenting the anomoly, not
changing behavior. Of course, I'm very conservative on this kind of
thing.

Is there a practical problem you experience, or does this
inconsistency just bother you?

Post by Marcus Harnisch
It doesn't occur to me why it would be difficult to return
(expand-file-name "/..")
=> "/"

As a matter of coding, it would be easy to make that change.

Post by Marcus Harnisch
(expand-file-name "/.")
=> "/"

That's what I get already. Even on systems where that is not an
absolute path that's well-defined, I think, although I haven't checked
that that's the result on Windows.

Post by Marcus Harnisch
/* `/../' is the "superroot" on certain file systems. */
Is there any file system still in use with this concept?

I believe they're referring to operating systems without unified file
name spaces (typically those which use devices not mounted on the file
hierarchy), which would include Windows.

If you want to pursue this, I think you need to ask on emacs-devel.

WDOT?

[Marcus Harnisch]

Post by Stephen J. Turnbull

Post by Marcus Harnisch
The good news is that at least this is FSF compatible behavior.

I think this is a killer argument for documenting the anomoly, not
changing behavior. Of course, I'm very conservative on this kind of
thing.
Is there a practical problem you experience, or does this
inconsistency just bother you?

It is easy enough to work around, I guess. In the task at hand I am
traversing a filesystem tree towards root via something along the
lines of:

(expand-file-name (concat dirname ".."))

Getting all the way to "/.." is a corner case situation that is
extremely unlikely to occur in practice, still needs to be dealt
with.

It was more my surprise than anything else that made me
investigate. Also bothering was the workaround suggested by Emacs'
docstring which simply isn't robust.

Post by Stephen J. Turnbull

Post by Marcus Harnisch
(expand-file-name "/.")
=> "/"

That's what I get already.

True. My mistake.

Post by Stephen J. Turnbull
If you want to pursue this, I think you need to ask on emacs-devel.

Not worth bothering. A docstring fix would be sufficient.

Regards
Marcus

[Stephen J. Turnbull]

Post by Marcus Harnisch

Post by Stephen J. Turnbull
If you want to pursue this, I think you need to ask on emacs-devel.

Not worth bothering. A docstring fix would be sufficient.

OK, thanks for following up!

Steve

[Marcus Harnisch]

Post by Marcus Harnisch

Post by Stephen J. Turnbull
If you want to pursue this, I think you need to ask on emacs-devel.

Not worth bothering. A docstring fix would be sufficient.

Not that I changed my mind about this (actually doing some
experimenting to find a suitable doc string amendment), but just for
the record:

(expand-file-name "/../foo/")
=> "/../foo/"

(expand-file-name "/../../foo/")
=> "/foo/"

(expand-file-name "/../../../foo/")
=> "/../foo/"

(expand-file-name "/../../../../foo/")
=> "/foo/"


(expand-file-name "/foo/../")
=> "/"

(expand-file-name "/foo/../../")
=> "/../"

(expand-file-name "/foo/../../../")
=> "/"

(expand-file-name "/foo/../../../../")
=> "/../"


I think you are getting the even/odd pattern. Beware of path names
that have parent references beyond root! This oscillating pattern is
particularly annoying when you traversing directory trees to find
project root directories or similar tasks.

Kindly
Marcus

[Stephen J. Turnbull]

Post by Marcus Harnisch
I think you are getting the even/odd pattern. Beware of path names
that have parent references beyond root! This oscillating pattern is
particularly annoying when you traversing directory trees to find
project root directories or similar tasks.

If (expand-file-name "/..") -> "/.." as currently[1], I see two ways
that we could go. One is to assume that in fact there is a superroot
(as on Windows, where there is a drive letter before the filesystem
root, but nothing before that, so (expand-file-name "/../..") -> "/.."
as well. The alternative would be to assume that it might be "turtles
all the way down," so (expand-file-name "/../..") -> "/../..",
(expand-file-name "/../../..") -> "/../../..", and so on. I suppose
you prefer the former, but feel free to tell me otherwise.

One thing we could potentially do would be to rethink the
functionality, decide what The Right Thing is based on 25 years of
experience with expand-file-name, implement that and call it
"canonicalize-file-name". I'm not sure that would be all that
beneficial, though, and I don't know that we have enough non-POSIX-
filesytem-semantics expertise lying around to decide what to do with
it.

Another possibility would be to see if we can hijack URL
canonicalization (which has well-known rules), but I suspect that
doesn't help with Windows, since Windows network servers don't expose
device names as far as I know.


Footnotes:
[1] For GNU Emacs compatibility.

[Mats Lidell]

Post by Stephen J. Turnbull
If (expand-file-name "/..") -> "/.." as currently[1], I see two ways
that we could go.

On the subject of twisty-little-functions-all-alike -- How does all of
this relate to file-truename which doesn't seem to have the flaw(!?)
as described above!?

Yours

[Stephen J. Turnbull]

Post by Mats Lidell
On the subject of twisty-little-functions-all-alike -- How does all of
this relate to file-truename which doesn't seem to have the flaw(!?)
as described above!?

file-truename actually follows the path on the device. See
realpath(1) and realpath(3) man pages. Unlike the system utilities,
however, file-truename only follows the actual path until it ends on
the device, then appends any components remaining in the given name.

expand-file-name, file-name-as-directory, and similar functions
manipulate strings, and so won't necessarily correspond to any real
directory entries (they might contain symlinks, as well as the
occasional anomolous "/..").

[Mats Lidell]

Post by Stephen J. Turnbull
file-truename actually follows the path on the device.

I'm confused (and possibly stupid!). (file-truename "/foo/bar/..") gives "/foo" on my system (XEmacs 21.5). I have no "/foo" directory on this machine. So what does "actually follow the path on the device" mean?

Yours

[Stephen J. Turnbull]

Post by Mats Lidell

Post by Stephen J. Turnbull
file-truename actually follows the path on the device.

I'm confused (and possibly stupid!). (file-truename "/foo/bar/..")
gives "/foo" on my system (XEmacs 21.5). I have no "/foo" directory
on this machine. So what does "actually follow the path on the
device" mean?

I'm not sure why this makes sense, but file-truename does
expand-file-name (or the equivalent in C) first. Then it eliminates
symlinks. So

/foo/bar/..
-> /foo # expand-file-name
-> / + foo # / exists and has realpath "/", foo does not exist
-> /foo # appends remaining components

or something like that (I don't have time to look at the code, but
it's pretty hairy IIRC).

[Mats Lidell]

Then it eliminates symlinks.

Ah. Got it. Thanks.

Yours

[Marcus Harnisch]

Post by Stephen J. Turnbull
If (expand-file-name "/..") -> "/.." as currently[1], I see two ways
that we could go. One is to assume that in fact there is a superroot
(as on Windows, [...]

I don't think the “superroot” in the comment refers to Windows, else
it would have been mentioned somehow. As a matter of fact, looking
over the fence revealed a bit more information[1]:

/* `/../' is the "superroot" on certain file systems.
Turned off on DOS_NT systems because they have no
"superroot" and because this causes us to produce
file names like "d:/../foo" which fail file-related
functions of the underlying OS. (To reproduce, try a
long series of "../../" in default_directory, longer
than the number of levels from the root.) */

Post by Stephen J. Turnbull
[...], but nothing before that,[...]

This is bothering me from a functionality perspective. Someone makes a
vague reference to file systems that may exist effectively as
subdirectories of a superroot, without giving a concrete example and
without stating why that superroot in turn couldn't be part of a
supersuperroot and so on. (I don't actually care, just curious)

Post by Stephen J. Turnbull
Another possibility would be to see if we can hijack URL
canonicalization (which has well-known rules), but I suspect that
doesn't help with Windows, since Windows network servers don't expose
device names as far as I know.

That wouldn't help in my case I am afraid, but is generally a good
idea.

In any case I'd neither change the function nor add a new one for this
one reason. We should just document that path names that may take you
outside of root are generally dangerous with this function and perhaps
refer them to #'file-truename to see if that gives better results.

Footnotes:
[1] http://git.savannah.gnu.org/cgit/emacs.git/tree/src/fileio.c

[Stephen J. Turnbull]

Post by Marcus Harnisch
This is bothering me from a functionality perspective. Someone makes a
vague reference to file systems that may exist effectively as
subdirectories of a superroot, without giving a concrete example and
without stating why that superroot in turn couldn't be part of a
supersuperroot and so on. (I don't actually care, just curious)

Doesn't Google like you? (Just teasing, you have no duty to check.)
On the first page of "filesystem superroot" results there are four
references to superroot that seem to make sense here. The first is
the obsolete Apollo Domain/OS filesystem, the second is Cygwin (which
presumably means it's embedded in the VFS, but that's just a WAG, I
haven't actually looked it up), the third is Linux NILFS, and the
fourth is the HAMMER2 filesystem associated with Dragonfly BSD.

The latter two are concrete filesystems, and apparently use the
superroot as a place to collect information about internal
filesystems, specifically snapshots in the case of HAMMER2. So for
these concrete filesystems, the "superroot" concept makes sense, and
it isn't "turtles all the way down".[1]

http://leaf.dragonflybsd.org/mailarchive/users/2012-02/msg00020.html
https://www.kernel.org/doc/Documentation/filesystems/nilfs2.txt

It's not clear to me that Emacsen have any business looking above root
in their ordinary operations, though. I would implement
expand-file-name consistently with POSIX semantics (since Emacs
*doesn't* implement "superroot" for DOS_NT filesystems!), and provide
a special superroot-directory function (or some such) for those who
really have need.

Regards,

Footnotes:
[1] http://evolvingthoughts.net/2011/03/turtles-all-the-way-down/ for
those who haven't seen the phrase before. And for those who have, and
have 30 minutes to browse, the comments are wonderful! (Some of them,
anyway. ;-)

[Marcus Harnisch]

Post by Stephen J. Turnbull
Doesn't Google like you? (Just teasing, you have no duty to check.)

Touché! Like I said I don't care enough since we can't realistically
change the behavior anyway. *If* we were going to change anything, I'd
skip all leading "/.." after each expansion. But this whole issue is
not even relevant enough to care, as long as we raise developers'
awareness with an appropriate docstring.

In my use case (traversing directories towards root) for instance the
trivial workaround is an abort condition which checks after every call
to #'e-f-n, whether the returned string is shorter than that of the
previous iteration.

Cheers
Marcus

[Jerry James]

On Wed, Jul 8, 2015 at 3:39 PM, Marcus Harnisch

Post by Marcus Harnisch
I don't think the “superroot” in the comment refers to Windows, else
it would have been mentioned somehow. As a matter of fact, looking
/* `/../' is the "superroot" on certain file systems.
Turned off on DOS_NT systems because they have no
"superroot" and because this causes us to produce
file names like "d:/../foo" which fail file-related
functions of the underlying OS. (To reproduce, try a
long series of "../../" in default_directory, longer
than the number of levels from the root.) */

So a Newcastle Connection kind of superroot, perhaps:

http://www.researchgate.net/publication/229793445_The_newcastle_connection_or_UNIXes_of_the_world_unite!

[Stephen J. Turnbull]

I guess that fits with the // notation for the superroot that the
Google blurb mentioning Cygwin mentions in the same breath.

(search terms: filesystem superroot)