Saturday, 19 October 2013

Mac OS X Lion, Windows and SMB interoperability. Seriously?

This post is about my ongoing woes with getting OS X Lion file sharing to play nice with Windows. It seems to me that SMB file sharing has always been broken in Lion for over a year now. The problems are subtle, but hinder significant functionality (such as Windows backup). So even though things basically work, there's always some -- not so rare -- case I run into, that requires odd tweaking, or simply can't be worked around.

Short background

I bought my first (and single) Mac machine, a mid-2012 Mac Mini, towards the end of 2012. Its intended purpose was to act as a home server capable of handling the following:

  • be small and silent
  • provide file sharing services (including networked windows backup)
  • have USB 3 allowing fast external disks to be attached
  • run three Linux virtual machines
  • run XBMC media player
  • have HDMI output for feeding 1080p signal to my TV

The Mac mini seemed to fit my requirements quite nicely, which prompted me to buy one. I must admit I haven't had any issues running virtual machines (using VirtualBox) with Linux. The mini also excels as a media player, happily running XBMC. Unfortunately though, when it comes to file sharing, the experience has been quite troublesome.

First attempt: the Linux issues and the SMB stack switch

I have an external USB3 drive attached to the mini, for which I enabled file sharing. To my dismay, I immediately ran into issues with Linux clients. Fortunately, the issue was quite easy to describe: mounting a share simply did not work and gave an error message along the lines of "mount error(126): Required key not available". This was easy to overcome, as searching for it quickly yielded results (basically it comes down to using some extra mount options: nounix,sec=ntlmssp).

It was at this point that I discovered that Apple had switched to their own "SMB stack" (the software responsible for network sharing with the Windows world) around the time I bought my mini. Other people were reporting this error, saying everything used to work but the OS X Lion update broke things. In the related threads, knowledgeable folk were explaining how the new OS X was using an SMB stack developed in-house by Apple.

The subtle Windows "huge file" issues

On Windows things were quite tricky. The problem was that seemingly everything was ok. You could browse shares, copy files, etc. At random points however, I would notice the Windows laptop get disconnected from the Mac. Initially, I didn't look into it, thinking it was a fluke. But then I started seeing a pattern of getting disconnected when accessing large files (e.g. saving a system image with Windows Backup, or working with a huge file like a Virtual Machine disk image, or watching video from a network share exported by the mini).

I assumed it was some networking issue and that the wireless adapter in the Mac wasn't playing well with my wireless hub. Searching on the internet (for all the wrong keywords, as it later turned out this was totally unrelated), I initially gave up and simply reverted to using a LaCie network drive for my Windows Backup.

The Windows "huge file" issue solution reveals more issues

I was happily using my LaCie network drive for windows, having given up on getting things fixed in OS X. After almost a year though, I noticed such a disconnection while logged into the Mac Mini with SSH. I realized then that the network adapter was fine, as the SSH session remained connected, so I decided to look into the issue again. This time I searched for issues with windows, smb disconnects and large files. Luckily, I soon found this Apple Discussions thread which provides a solution!

It turns out that the way Apple has implemented their stack, causes Windows to not be able to calculate a proper session timeout for the exchange. Anyway, I happily applied the advice in that post and edited the Windows registry to increase the session timeout.

Now, this indeed did improve things with huge files, so I thought I'd try a Windows Backup to the mini again. After all, by now I had a RAID-10 volume attached to the mini and I really wanted my backups to be in the safest place possible (the LaCie network drive was nearly 7 years old).

Unfortunately, I now started getting a weird error code 0x81000039 when trying to access the share via Windows Backup. Searching for this yielded this solution which involves editing the Windows security policy. In this case I can't really blame Apple as this seems to be an incompatibility among various versions of windows as well.

Back to square one: the "read-only attribute" fiasco

So, after a year of tinkering I had a mini that was happily sharing files (even large ones) without disconnecting and without the odd access issues and weird error codes. I once again tried to do a network backup and... surprise! It still didn't work. This time the backup would start, create the folder in the network share and then fail with an access error. Looking at the created folder with finder, I found that the folder was "locked", which in OS X Lion means it has the uchg flag set.

Digging into this I discovered another person had just run into this same issue which was due to a recent update in Lion, which involves setting this immutable flag when an SMB operation is made to make a folder read-only. On Windows, the read-only flag is handled as merely an indication that the folder should not be messed with, as it is used for system purposes. Windows Backup sets this flag on the top directory it creates during backup. As a result, when it then goes on to create files in it, Mac OS X doesn't let it (whereas Windows system do).

Stick with what you know works

At this point I was disappointed once again. There was nothing I could do about this, except wait for an update from Apple that reverts to the previous behavior. I had lost a lot of time trying to get things to work, to no avail. Furthermore, with so many changes by Apple, I no longer had any faith that whatever hack someone comes up with will result in a "stable" solution: the next OS X update could easily break things again. My only native option was to install regular Samba via MacPorts, but then I'd lose the ability to configure things via the System Preferences user interface. By now everything else worked, I just needed to get a file share that would keep Windows Backup happy. Finally, I decided to go back to what I know well and trust: Samba on Linux.

Like I mentioned, the mini is also running VirtualBox with Linux virtual machines. I used VirtualBox's shared folder feature to mount the external drive under a Linux VM, then exported that location from within Linux using regular Samba. I'm finally able to backup my Windows laptop on the RAID-10 volume attached to the mini and retire my ageing LaCie network drive!

Sunday, 26 June 2011

Integrating: Jetty, Spring, AspectJ, Wicket, Hibernate, MySQL, ActiveMQ and Atomikos (web archive scope)

Overview


The Spring framework has long provided an attractive alternative to J2EE application servers (especially prior to the arrival of JavaEE 6). Development teams often opted to go for a setup that uses Spring to manage the application context, coupled with a plain Servlet/JSP container and a good ORM solution.

Indeed, the "Jetty/Tomcat + Spring + Hibernate" stack has served many projects. As growth and complexity kicks in, things like JMS and (not long after that) XA transactions, start becoming a necessity.

It is at that point that teams start discussing the use of an application server, in order to gain a "certified" stack that combines all of the above (and more) facilities. However, some teams want to keep on using the "roll your own" path, by adding these aspects into the mix.

In this entry I will describe a setup that combines the most common JEE APIs using a custom assembly of components. The setup comprises of:

  • Jetty 7.3.1 (Servlet/JSP)
  • Spring 3.0.5 (CDI)
  • AspectJ 1.6.10 (because it is cool)
  • Atomikos 3.7.0 (JTA with XA support)
  • Hibernate 3.6.0 (JPA)
  • ActiveMQ 5.5.0 (JMS)
  • MySQL (JDBC, using driver 5.1.15)

Note that all of the above are integrated at the Web Archive level (i.e. the libraries are all included in WEB-INF/lib with nothing placed in the JVM classpath). This is because I often choose to "embed" Jetty when working in this mode, using my own custom launcher.

The basics: a Spring-enabled WAR, with Wicket managing the presentation


Since we will use Spring to inject all of the resources (JTA, JDBC connections, JPA Persistence Units, JMS, ...), the first step is to create a Spring-enabled WAR. In this step, we will also integrate Apache Wicket to take over page generation (sorry, but I have long hated JSF and JSP with a passion, so these are the first thing I want to get rid of.

So, we start by updating our web.xml as follows:




 My own near-JEE stack

 
  Wicket mode (development/deployment)
  wicket.configuration
  
  
  development
 

 
  my.app.filter.wicket
  org.apache.wicket.protocol.http.WicketFilter
  
   applicationClassName   my.Application  
 
 
  
  my.app.filter.wicket
  /*
 

 
  Main Spring context configuration file
  contextConfigLocation  
  /WEB-INF/applicationContext.xml
 

 
  
  org.springframework.web.context.ContextLoaderListener
 

 
  BASIC
 


This is all that is needed for web.xml. The rest of the magic lies totally within Spring's configuration file (/WEB-INF/applicationContext.xml).




 
 

 
 

 
 

 
 



Note that the references to META-INF/spring/... are presented in the following sections.

Adding the Atomikos JTA implementation


The next step is to create the META-INF/spring/tx-services.xml where we instantiate the Atomikos transaction manager and bind it as our JTA implementation. Note that this is where we need to explaing to Spring that we are using AspectJ and therefore the @Transactional annotation must be handled via @Configurable at the class implementation level.




 
 
  
  
 
 
 
  
 

 
 
  
  
 

    
        
    

    
    


Integrating ActiveMQ as a JMS provider that uses Atomikos for transaction management


The next step is to create the META-INF/spring/jms-services.xml where we define the ActiveMQ broker and tell it to interact with Atomikos for participating in transactions...




 
 
  
  
         
    
   
  
  
   
    
    
    
    
    
    
   
  
 

 
 
  
  
  
 

 
  
   
  
  
   
  
 

 

 
  
  
 

 

 
  
 


Integrating Hibernate as a JPA provider that uses Atomikos for transaction management


Last but not least, the META-INF/spring/db-services.xml will add Hibernate to the mix, telling it to interact with Atomikos for participating in transactions.

Important Note 1: that I have a lot of ${database.variable} that I usually load from a property file; you should replace this with whatever values are suitable for your case.

Important Note 2: The hibernate.transaction.factory_class is explicitly shown as commented out, as I have seen quite a few people using it (due to picking up old Atomikos documentation) when it is NO LONGER NEEDED for things to work properly (see the Atomikos forum thread referenced in the comment).




 
  XADBMS
  
   com.mysql.jdbc.jdbc2.optional.MysqlXADataSource
  
  
   
    ${database.url}
    ${database.username}
    ${database.password}
   
  
  
  
  

 

 
 
  
   
  
 

 
    
        
        
        
            
                
                
                
            
        
        
            
                
                 org.hibernate.cfg.ImprovedNamingStrategy
                
                
                 ${database.hibernate.hbm2ddl}
                
                
                
                    com.atomikos.icatch.jta.hibernate3.TransactionManagerLookup 
                 
            
        
    


Important Note 3:Remember to use JTA mode for transactions in your persistence.xml, as in:


...

Friday, 24 June 2011

Spring exception translation with AspectJ: bug workaround

One of the nice features of Spring is the database access exception translation that simplifies error-handling. Most of us enabled it by adding the PersistenceExceptionTranslationPostProcessor (see the Exception translation in the Spring reference).

Unfortunately, when using a "pure-AspectJ" approach, you will find that the out-of-the-box JpaExceptionTranslatorAspect does not recognize the various JPA dialects out there. I was pretty dissappointed when I found that JPA access from @Configurable object instances does NOT undergo "proper exception translation".

Thanks to Martin Deinum's feedback I quickly "restored" this limitation by creating a custom aspect to invoke the "proper" exception translator (which in my case is the HibernateJpaDialect).

So, until this bug is resolved, the following aspect will take care of things:

import org.springframework.dao.DataAccessException;
import org.springframework.orm.jpa.vendor.HibernateJpaDialect;

public aspect HibernateJpaDialectExceptionTranslationAspect {
    pointcut entityManagerCall(): call(* javax.persistence.EntityManager.*(..)) || call(* javax.persistence..EntityManagerFactory.*(..)) || call(* javax.persistence.EntityTransaction.*(..)) || call(* javax.persistence.Query.*(..));
    
    after() throwing(RuntimeException re): entityManagerCall() {
     HibernateJpaDialect hibernateDialect = new HibernateJpaDialect();
     DataAccessException dex = hibernateDialect.translateExceptionIfPossible(re);

     if (dex != null) {
      throw dex;
     } else {
      throw re;
     }        
    } 
}

Sunday, 12 June 2011

JDeveloper 11g 11.1.2 is out with Maven support (among others)

Oracle have released Oracle JDeveloper 11g, version 11.1.2.0.0. Among other new features, such as JSF 2, my favorite is the Maven 2 integration!

Check it out! Download is available (as usual) from the technet site!

IMPORTANT NOTE: SOA & WebCenter are NOT yet available for this release, so you must stick 11.1.1.5 for such projects.

Friday, 27 May 2011

Using Beehive code (BEA WebLogic <=10.3.1) with recent versions of Oracle WebLogic

Recent versions of the WebLogic Server no longer ship with Beehive libraries. As a result, old code that was based on Beehive must ne re-written so as to remove this dependency.

If you simply must deploy a Beehive project on a newer WebLogic Server, there is way to achieve that. Oracle will NOT support it of course, but if you decide to take the risk and go ahead with this anyway, here is how to do it:

  1. Locate the library "wlw-system.jar" in your old WebLogic installation. Copy it to the library folder of your target domain (the one that is based on a newer WebLogic version that does not have Beehive). The folder is ${DOMAIN_HOME}/lib
  2. Restart your domain's servers (in order to pick up the wlw-system.jar in their classpath)
  3. Using workshop, look up the WebLogic shared libraries that your project uses. These are listed in the project properties at "Java Build Path --> Libraries"
  4. For each shared library go to your Workshop preferences under "WebLogic-->Shared Libraries" and click "Edit.." to see the location of the library. Copy all of the EAR/WAR archives for these libraries into some "staging" folder (e.g. /home/yourusername/app-libs)
  5. Deploy all of the archives (EAR/WAR) that you located in the steps above as shared libraries in your new domain

You may now deploy your application on servers where the shared libraries were targeted.

Wednesday, 25 May 2011

Using a different HTTP proxy per each JAX-WS client stub

JAX-WS is about to standardize a web service feature, for specifying the use of a different HTTP proxy per each JAX-WS client instance. This can be extremely useful in complex enterprise IT environments, where network security teams are often paranoid and have different proxies for accessing different parts of the network. No longer do you need to host applications modules on different VMs just because they require different proxy settings. Instead of using the global http.proxyHost and http.proxyPort, you can specify a different proxy per JAX-WS client stub.

Since OFM 11gR1 Patchset 2, one can already exploit this upcoming JAX-WS feature, using WebLogic's ClientProxyFeature API. The approach is described in WebLogic's API Reference.

Although the documentation is pretty straight-forward, it fails to address the fact that the JAX-WS stack will try to retrieve the WSDL during the service instantiation, which leads to the following problem: since the ClientProxyFeature only applies at the web service port level, the following will fail:

// The service location is a runtime configuration value...
    URL wsdlURL = new URL(System.getProperty("serviceLocationAtRuntime"));

    // The below will try to fetch the WSDL from the wsdlURL location
    // The proxy is NOT in effect at this point and will cause a network error
    Service myService = new Service(wsdlURL, ...);

    // The code will never reach this point; so much for ClientProxyFeature
    ClientProxyFeature cpf = new ClientProxyFeature();
    cpf.setType(Proxy.Type.HTTP);
    cpf.setProxyHost(CONFIG.getProxyHost());
    cpf.setProxyPort(CONFIG.getProxyPort());
    servicePort = myService.getServiceSoap(cpf);

The solution is to bundle the WSDL as a resource in your application and instantiate the service using the local copy of the WSDL. Then, after enabling the proxy, you can set the runtime service address via the BindingProvider API.

// Don't go over the network to fetch the WSDL; use it from the classpath
    URL wsdlURL = Service.class.getResource("Service.wsdl");

    // The below will work, but will assume the service address is whatever
    // was defined in your application-bundled WSDL, so we still need to
    // somehow define a run-time location; keep reading to see how we do this
    // after getting the service port
    Service myService = new Service(wsdlURL, ...);

    // Enable the use of an HTTP proxy for this specific port instance
    ClientProxyFeature cpf = new ClientProxyFeature();
    cpf.setType(Proxy.Type.HTTP);
    cpf.setProxyHost(CONFIG.getProxyHost());
    cpf.setProxyPort(CONFIG.getProxyPort());
    servicePort = myService.getServiceSoap(cpf);

    // Now that we have the port, configure the service address using
    // a run-time value:
    URL wsdlURL = new URL(System.getProperty("serviceLocationAtRuntime"));
    BindingProvider bp = (BindingProvider)servicePort;
    bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
        wsdlURL.toString());

    // You can now call the web service via the HTTP proxy
    servicePort.invokeSomeOperation(...);

Wednesday, 18 May 2011

Mental note to myself # 8734874243

Oracle SOA Suite 11g -- Repository notes


...because I always forget, let me put these down for my future reference!

Status values for composite instances



The following values are used for the CUBE_INSTANCE.STATE column of SOA Suite's "SOAINFRA" schema.

OPEN states CLOSED states
INITIATED PENDING_CANCEL
RUNNING COMPLETED
SUSPENDED FAULTED
FAULTED CANCELED
ABORTED
STALE

Some SQL to quickly review statistics regarding the state of composite instances within some time window:

SELECT
  CASE state
    WHEN 0 THEN 'initiated'
    WHEN 1 THEN 'open.running'
    WHEN 2 THEN 'open.suspended'
    WHEN 3 THEN 'open.faulted'
    WHEN 4 THEN 'closed.pending_cancel'
    WHEN 5 THEN 'closed.completed'
    WHEN 6 THEN 'closed.faulted'
    WHEN 7 THEN 'closed.cancelled'
    WHEN 8 THEN 'closed.aborted'
    WHEN 9 THEN 'closed.stale'
    ELSE 'huh?'
  END state_text,
  COUNT (*)
FROM cube_instance ci
WHERE ci.creation_date >= (:startdate) AND ci.creation_date < (:enddate)
group by ci.state;

Status of “recently created” composite instances


What's going on with recently created instances?
SELECT
  CIKEY, STATE, CREATION_DATE, MODIFY_DATE, STATUS, 
  COMPOSITE_NAME, COMPONENT_NAME, COMPOSITE_REVISION, DOMAIN_NAME
FROM cube_instance 
WHERE creation_date >= SYSDATE-1 AND STATE < 5 AND ROWNUM < 20
ORDER BY creation_date DESC;

Status of “recently updated” composite instances


What's going on with recently updated instances?
SELECT
  CIKEY, STATE, CREATION_DATE, MODIFY_DATE, STATUS,
  COMPOSITE_NAME, COMPONENT_NAME, COMPOSITE_REVISION, DOMAIN_NAME
FROM cube_instance 
WHERE creation_date >= SYSDATE-1 AND STATE < 5 AND ROWNUM < 20 
ORDER BY modify_date DESC;