OMERO image import using J2EE
Aug. 25th, 2018 02:22 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
mtbcDespite my tiredness I seem to remain effective at my day job. Last week mostly I focused on improving the process by which microscope images are transferred from client software into the OMERO server. This is a complex process with many substeps and variables depending on configuration and the kind of image data being imported.
I was largely productive but did run into a couple of interesting issues. One is that I had found my way to Java's ExecutorService class but would prefer to be notified when any of the submitted threads complete without having to poll them. I now find that the easiest solution may be to use ExecutorCompletionService so that will be a task for Monday.
Another issue is that for writing the imported image files on the server, together with parent directories, the import is running in the context of a single database transaction and our file storage has the actual filesystem objects but also corresponding ownership metadata in a database table that allows OMERO's permissions system to determine quickly who may access the file. The server complains when there is a database row or a filesystem object but not both.
There is a race condition when multiple imports need to create the same parent directory. One thread may create the filesystem object and write the ownership information. Before that thread's transaction commits, a different thread may detect that the filesystem object already exists and, thanks to isolation, not see the corresponding ownership information anytime soon. So far an easy but robust solution to this has not come to my mind. I do provide the workaround of allowing users to configure the server so that image data is written within a parent directory named for the server thread that is managing the import.
I was largely productive but did run into a couple of interesting issues. One is that I had found my way to Java's ExecutorService class but would prefer to be notified when any of the submitted threads complete without having to poll them. I now find that the easiest solution may be to use ExecutorCompletionService so that will be a task for Monday.
Another issue is that for writing the imported image files on the server, together with parent directories, the import is running in the context of a single database transaction and our file storage has the actual filesystem objects but also corresponding ownership metadata in a database table that allows OMERO's permissions system to determine quickly who may access the file. The server complains when there is a database row or a filesystem object but not both.
There is a race condition when multiple imports need to create the same parent directory. One thread may create the filesystem object and write the ownership information. Before that thread's transaction commits, a different thread may detect that the filesystem object already exists and, thanks to isolation, not see the corresponding ownership information anytime soon. So far an easy but robust solution to this has not come to my mind. I do provide the workaround of allowing users to configure the server so that image data is written within a parent directory named for the server thread that is managing the import.
