This post is about my ongoing woes with getting OS X Lion file sharing to play nice with Windows. It seems to me that SMB file sharing has always been broken in Lion for over a year now. The problems are subtle, but hinder significant functionality (such as Windows backup). So even though things basically work, there's always some -- not so rare -- case I run into, that requires odd tweaking, or simply can't be worked around.
I bought my first (and single) Mac machine, a mid-2012 Mac Mini, towards the end of 2012. Its intended purpose was to act as a home server capable of handling the following:
- be small and silent
- provide file sharing services (including networked windows backup)
- have USB 3 allowing fast external disks to be attached
- run three Linux virtual machines
- run XBMC media player
- have HDMI output for feeding 1080p signal to my TV
The Mac mini seemed to fit my requirements quite nicely, which prompted me to buy one. I must admit I haven't had any issues running virtual machines (using VirtualBox) with Linux. The mini also excels as a media player, happily running XBMC. Unfortunately though, when it comes to file sharing, the experience has been quite troublesome.
First attempt: the Linux issues and the SMB stack switch
I have an external USB3 drive attached to the mini, for which I enabled file sharing. To my dismay, I immediately ran into issues with Linux clients. Fortunately, the issue was quite easy to describe: mounting a share simply did not work and gave an error message along the lines of "mount error(126): Required key not available". This was easy to overcome, as searching for it quickly yielded results (basically it comes down to using some extra mount options: nounix,sec=ntlmssp).
It was at this point that I discovered that Apple had switched to their own "SMB stack" (the software responsible for network sharing with the Windows world) around the time I bought my mini. Other people were reporting this error, saying everything used to work but the OS X Lion update broke things. In the related threads, knowledgeable folk were explaining how the new OS X was using an SMB stack developed in-house by Apple.
The subtle Windows "huge file" issues
On Windows things were quite tricky. The problem was that seemingly everything was ok. You could browse shares, copy files, etc. At random points however, I would notice the Windows laptop get disconnected from the Mac. Initially, I didn't look into it, thinking it was a fluke. But then I started seeing a pattern of getting disconnected when accessing large files (e.g. saving a system image with Windows Backup, or working with a huge file like a Virtual Machine disk image, or watching video from a network share exported by the mini).
I assumed it was some networking issue and that the wireless adapter in the Mac wasn't playing well with my wireless hub. Searching on the internet (for all the wrong keywords, as it later turned out this was totally unrelated), I initially gave up and simply reverted to using a LaCie network drive for my Windows Backup.
The Windows "huge file" issue solution reveals more issues
I was happily using my LaCie network drive for windows, having given up on getting things fixed in OS X. After almost a year though, I noticed such a disconnection while logged into the Mac Mini with SSH. I realized then that the network adapter was fine, as the SSH session remained connected, so I decided to look into the issue again. This time I searched for issues with windows, smb disconnects and large files. Luckily, I soon found this Apple Discussions thread which provides a solution!
It turns out that the way Apple has implemented their stack, causes Windows to not be able to calculate a proper session timeout for the exchange. Anyway, I happily applied the advice in that post and edited the Windows registry to increase the session timeout.
Now, this indeed did improve things with huge files, so I thought I'd try a Windows Backup to the mini again. After all, by now I had a RAID-10 volume attached to the mini and I really wanted my backups to be in the safest place possible (the LaCie network drive was nearly 7 years old).
Unfortunately, I now started getting a weird error code 0x81000039 when trying to access the share via Windows Backup. Searching for this yielded this solution which involves editing the Windows security policy. In this case I can't really blame Apple as this seems to be an incompatibility among various versions of windows as well.
Back to square one: the "read-only attribute" fiasco
So, after a year of tinkering I had a mini that was happily sharing files (even large ones) without disconnecting and without the odd access issues and weird error codes. I once again tried to do a network backup and... surprise! It still didn't work. This time the backup would start, create the folder in the network share and then fail with an access error. Looking at the created folder with finder, I found that the folder was "locked", which in OS X Lion means it has the uchg flag set.
Digging into this I discovered another person had just run into this same issue which was due to a recent update in Lion, which involves setting this immutable flag when an SMB operation is made to make a folder read-only. On Windows, the read-only flag is handled as merely an indication that the folder should not be messed with, as it is used for system purposes. Windows Backup sets this flag on the top directory it creates during backup. As a result, when it then goes on to create files in it, Mac OS X doesn't let it (whereas Windows system do).
Stick with what you know works
At this point I was disappointed once again. There was nothing I could do about this, except wait for an update from Apple that reverts to the previous behavior. I had lost a lot of time trying to get things to work, to no avail. Furthermore, with so many changes by Apple, I no longer had any faith that whatever hack someone comes up with will result in a "stable" solution: the next OS X update could easily break things again. My only native option was to install regular Samba via MacPorts, but then I'd lose the ability to configure things via the System Preferences user interface. By now everything else worked, I just needed to get a file share that would keep Windows Backup happy. Finally, I decided to go back to what I know well and trust: Samba on Linux.
Like I mentioned, the mini is also running VirtualBox with Linux virtual machines. I used VirtualBox's shared folder feature to mount the external drive under a Linux VM, then exported that location from within Linux using regular Samba. I'm finally able to backup my Windows laptop on the RAID-10 volume attached to the mini and retire my ageing LaCie network drive!