Takin
Takin by Tobias Weber
Bundle contents
Takin by Tobias Weber is a fully written in C++11 and relies on pure Qt5 for its GUI. Tobias I does not really use an IDE, but vi, nano and kate and gdb as debuger. Sometimes he also opens it in VSCode. The bundle generated by Tobias is not directly suitable for a Mac App Store submission.
The initial bundle is quite complicated (13200 files) with nested bundles, i.e. an embedded Python. It contains:
231 libraries
12 auxiliary binaries
1 embedded bundle (Phyton.app)
Folder "Contents"
Frameworks <-- many .dylib, .so, .a, .o libraries, some binaries, embedded bundle "Python.app" and even hidden files and folders
Info.plist <-- several missing keys
Libraries
MacOS
PlugIns <-- many libraries
res
Resources
site-packages
Testing auxiliary binaries
Actions to activate the auxiliary binaries:
takinmod_py (ID: eu.ill.cs.takin.modpy): Resolution / Convolution… —> Convolution dialog —> "Model Source” —> does the entry "Python Model" appear ?
This is the only use of Python
It creates a shared memory through "Boost.interprocess"
Resolution / Convolution..." and from the dialog's "File" menu (not the one form the main window) choose "Load..." and select the file "phonon_simple_py.taz". It will crash, and write the following into the log (accessible from the "Help" menu):
takin_pol (ID: eu.ill.cs.takin.pol): Tools / Polarisation Vectors… —> displays a window.
takin_cif2xml (ID: eu.ill.cs.takin.cif2xml): "File / Import CIF...". Select a CIF file to see if it works.
takin_structfact (ID: eu.ill.cs.takin.structfact): "Tools / Nuclear Structure Factors..."
takin_magstructfact (ID: eu.ill.cs.takin.magstructfact): "Tools / Magnetic Structure Factors..."
Share memory test
"Resolution" -> "Convolution..." and choose "File" -> "Load...", e.g. files from "phonon_simple_py.taz"
the file has to be loaded in the convolution dialog, not in the Takin main window
For this to work, the python module has to successfully load, and shared memory creation must work.
If everything works, you should be able to click on --> "Start Sim."
a resolution convolution of a phonon peak should start.
If nothing happens, the python module could not start.
You can start takin from the command line and will get some logs.
List of file changes
The bundle generated by the tools that Tobias uses is not directly ready for submission to the App Store.
Thus a list of changes must to be performed prior signing, packaging and submiting it to iTunes Connect.
Architectures
- i386: this one is now forbidden and must be stripped of from all binaries and libs.
- arm64: Takin.app is x86_64 compatible but only some files are arm64 compatible. Thus arm64 should be strippd of.
Removing unnecessary files
The number of libraries was progressively reduced to 225 (.a, .so, .dylib) + one library with no extension.
Removed: all Tcl and Tk libs and associated files.
Removed: Python.app. A bundle which depends on external libs and hence was not functional.
Removed: python3.9. A binary which depends on external libs and hence was not functional.
Removed: python.o a file which has no LC_UUID
Removed: some hidden files/folders and a Python cache __pycache__.
Takin.app bundle
Icon file takin.icns
It must contain two icons 512x512 and 1024x1024 in size.
Info.plist
Modified key e.g.: CFBundleIdentifier
Additional keys e.g.: CFBundleVersion, CFBundleDevelopmentRegion, LSApplicationCategoryType, LSMinimumSystemVersion, ITSAppUsesNonExemptEncryption
Caution: with Big Sur, spctl may complain that Info.plist has been modified if it not in the binary format.
Python.app
Icon file: PythonInterpreter.icns
The icon of the original bundle is 128x128 only. However, Apple does not reject it.
Info.plist
Modified key: CFBundleIdentifier otherwise it wont sign.
In fact, this bundle is not functional due to missing libraries. Moving inside the bundle:
Takin.app/Contents/Frameworks/Python.framework/Versions/3.9/Python
and setting up an "rpath" is not enough to solve the problem.
Fortunately enough Python.app is not used by Takin.app and can trashed.
Code signing and sandboxing
Files to be signed and sandboxed:
- /MacOS/binaries
- auxiliary binaries present in /Frameworks
- libraries that are not already signed (.a, .so, .dylib, .o, and without an extension)
- the bundle itself
1-3: to be signed prior to the bundle,
4: do not use the "codesign -deep", Apple says its not recommanded
Checking signature
$ codesign --verify --deep --strict --verbose=4 Takin.app
This works whatever the certificate used.
Checking the bundle for the App Store
$ spctl --assess --verbose=4 Takin.app
The answer is a mere "Accepted" or "Rejected" with no more explanation.
Mind the certificate: the answer will always be "rejected" for bundle signed with the "Apple distribution" certificate!
Certificate: "Developer ID"
Bundle signed
Takin4Mac-noSand.app: accepted
source=Notarized Developer ID
Bundle signed/sandboxed
Takin4Mac.app: accepted
source=Notarized Developer ID
Certificate: "Apple distribution"
Bundle signed/sandboxed
Takin.app: rejected
$ spctl --assess --raw --verbose=4 Takin.app
Mind the certificate, if it is "Apple distribution" a valid bundle will give the wrong message below:
Takin.app: a sealed resource is missing or invalid
Here a typical answer with certificat "Developer ID":
Takin.app: rejected
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>assessment:authority</key>
<dict>
<key>assessment:authority:flags</key>
<integer>0</integer>
</dict>
<key>assessment:remote</key>
<true/>
<key>assessment:verdict</key>
<false/> <--- ??? but why ???
</dict>
</plist>
But why ?: The reasons can be:
* a forbidden i386 architectures
* libs or bin with no LC_UUID
* a missing sealed ressource
* etc.
Problems and solutions
* Upload errors
ITMS-90236: Missing required icon
Use the command "iconutil" to create an .icns file containing both a 512x512 and a 1024x1024 icon. The file can be checked with Preview.app.
ERROR ITMS-90240: "Unsupported Architectures : i386
takin.app/Contents/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/.dylibs/libgcc_s.1.dylib
takin.app/Contents/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/.dylibs/libgcc_s.1.dylib
lipo libgcc_s.1.dylib -remove i386 -output /Users/afilhol/Desktop/libgcc_s.1_new.dylib
ERROR ITMS-90255: "The installer package includes files that are only readable by the root user.
Update of permissions for the whole package content using Finder "Get Info".
Transporter says: "Failed to decompress the app."
This should be followed by list of reasons but with Takin.app I got none.
One should understand "Transporter successfully decompressed the App, but the bundle was rejected because ..."
After a lot of efforts, it turned out that:
- a file "python.o" had a missing LC_UUID (see below), Fortunately it could be deleted.
- one of the lib "Python" with no extension had a "missing sealed ressource". It had to be signed again since some unused files had been removed from the folder (see Sealed Resources).
* Embedded Python
Tobias initially used the HomeBrew distribution. Unfortunately it turned out that this Python is only suitable for local use. Some libraries cannot be signed or lack the Minimum OS Version key LC_VERSION_MIN_MACOSX, etc.
The solution was to use the official Python distribution instead: https://www.python.org/downloads/mac-osx/ The later is already partially ready for the arm64 architecture but neither libraries such as numpy and scipy nor the binaries. Thus the arm64 architecture can be safely stripped of.
* Non-public framework
Your app links against the following non-public framework(s):
Private APIs found:
Contents/Frameworks/Python.framework/Versions/3.9/lib/libtk8.6.dylib/NSDrawerWindow
Message to the "Resolution Center"
Dear reviewers,
We were surprised of that problem which is far beyond our control since we are mere users of TK libraries.
Anyway, we think that this cannot have any impact on the user experience for the reasons listed below.
So, please, accept our application for the App store.
All the best
The Takin team
1- NSDrawerWindow : API not used by Takin
Takin is fully written in C++11 and relies on pure Qt5 for its GUI, and itself does not use 'NSDrawerWindow'.
The only place that links against this symbol seems to be a library from the bundled Python distribution, which is the original one from python.org.
2- NSDrawerWindow cannot lead to a poor user experience
If, in the worst case, the bundled Python interpreter stopped working because of an external API change, this would only disable the optional Python plug-in module in Takin, but not affect the main programme.
3- NSDrawerWindow is a one line of code subroutine
The source code is here <http://svn.python.org/projects/external/tk-8.5.15.0/macosx/tkMacOSXWm.c>
@interface NSDrawerWindow : NSWindow
{ id _i1, _i2; }
@end
4- Why NSDrawerWindow amongst 608 other symbols ?
nm -gU Takin.app/Contents/Frameworks/Python.framework/Versions/3.9/lib/libtk8.6.dylib/NSDrawerWindow
This lists 607 other symbols. Thus it seems quite odd that your check tool identifies the only NSDrawerWindow as a private API.
Stupid answer from the Resolution Center
Thank you for your response and patience. However, we have still found the Private API found:
• Contents/Frameworks/Python.framework/Versions/3.9/lib/libtk8.6.dylib/NSDrawerWindow
Solution
Both TK and TCL comes with the Pyton distribution but are not used by Takin. We removed all the related libraries and files.
* Loading A Python module
If you select "Resolution" -> "Convolution..." in the menu, it fails to load the python plug-in module (takinmod_py) with the message:
"Unable to get bundle identifier for container id takinmod_py:
Unable to get bundle identifier because code signature information has no Info.plist.
Note that "takinmod_py" is one of the 6 auxiliary binaries located in the MacOS folder. The other banaries are in the Frameworks folder.
codesign --display --verbose=4 takin.app/Contents/MacOS/takin
Executable=/Users/afilhol/Development/Dev-Takin/_App-Store/Takin-full/test/takin.app/Contents/MacOS/takin
Identifier=eu.ill.cs.takin
Info.plist entries=18
codesign --display --verbose=4 takin.app/Contents/MacOS/takinmod_py
Executable=/Users/afilhol/Development/Dev-Takin/_App-Store/Takin-full/test/takin.app/Contents/MacOS/takinmod_py
Identifier=takinmod_py <--- default identifier
Info.plist=not bound <--- how to bound it ?
Can we solve this problem with the -i option of codesign ?
codesign -i eu.ill.cs.takin ...
This codesign option can be used to specify an identifier for each binary one by one.
- auxiliary binaries located in folder MacOS : they have no attached Info.plist and hence a default ID.
If we use the -i option to attach them an ID prior to signing the whole package, this will not work since the final signing will reset the ID to default - auxiliary binaries located in folder Framework : they have and attached Info.plist and even Info.plist files.
If the -i option is used for these binaries the App Store checks will detect an inconsistency between the assigned ID (e.g. eu.ill.cs.takin) and the recorded ID (e.g. org.qt-project.QtSvg) and will reject the bundle. - Bundle signing with the -i option: this will will set a unique ID to all the banaries.
The bundle will be rejected since Framework binaries will exhibit ID inconsitencies.
The solution (?): Injecting an Info.plist to /MacOS/ binaries
<https://developer.apple.com/library/archive/documentation/Security/Conceptual/CodeSigningGuide/Procedures/Procedures.html#//apple_ref/doc/uid/TP40005929-CH4-SW6>
Adding an Info.plist to Single-File Tools <-- would be perfect but this is for Xcode only
<https://twocanoes.com/adding-a-command-line-tool-helper-to-a-mac-app-store-app/>
Tell GCC to inject an Info.plist into a binary during compilation by including a CC variable:
./configure CC=”gcc -sectcreate,__TEXT,__info_plist,/path/to/my-info.plist”
As a result the command "otool -l" quotes:
Section
sectname __info_plist
segname __TEXT
addr 0x0000000100033846
size 0x00000000000001d9
offset 211014
align 2^0 (1)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Sandbox: takin_pol(3248) deny(1) forbidden-sandbox-reinit
The ID the auxiliary binaries is now Okay but the sandboxing setting is not. While all auxiliary binaries are set to "inherited", the final bundle codesign with the "-deep" option resets all to the bundle entitlements and changes the entitlements (see below). The "-deep" was necessary because of the embedded bundle "Python.app". Fortunately enough this bundle "Python.app" could be removed.
* LC_UUID
Here the error reported by the excellent "App Wrapper 4"
"Contents/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin/python.o" x86_64 requires a LC_UUID; Notarization or App Store submission may fail until this is corrected.
LC_UUID: a unique random UUID, usually produced by the static linker.
ld linker: -no_uuid Do not generate an LC_UUID load command in the output file.
<https://stackoverflow.com/questions/10119700/how-to-get-mach-o-uuid-of-a-running-process>
$ otool -l Takin.app/Contents/MacOS/takin | grep LC_
Load command 8
cmd LC_UUID
cmdsize 24
uuid BCBACEFB-1333-3274-B9BD-A0958FB76930 <-- this is the LC_UUID
Comparison:
$ otool -l Takin.app/Contents/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/.dylibs/libgcc_s.1.dylib | grep LC_
cmd LC_SEGMENT_64
cmd LC_SEGMENT_64
cmd LC_SEGMENT_64
cmd LC_ID_DYLIB
cmd LC_DYLD_INFO_ONLY
cmd LC_SYMTAB
cmd LC_DYSYMTAB
cmd LC_UUID <--- compulsory for the App Store
cmd LC_VERSION_MIN_MACOSX
cmd LC_SOURCE_VERSION
cmd LC_LOAD_DYLIB
cmd LC_FUNCTION_STARTS
cmd LC_DATA_IN_CODE
cmd LC_DYLIB_CODE_SIGN_DRS
cmd LC_CODE_SIGNATURE
$ otool -l Takin.app/Contents/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin/python.o | grep LC_
cmd LC_SEGMENT_64
cmd LC_VERSION_MIN_MACOSX
cmd LC_SYMTAB
cmd LC_DYSYMTAB <-- no LC_UUID found
* a sealed resource is missing or invalid
Cleaned bundle Takin.app
codesign --display --verbose=4 Takin.app/Contents/Frameworks/Python.framework/Versions/3.9/Python
Sealed Resources version=2 rules=13 files=5482 <-- but only 199 found and listed as "Modified file"
This means that "_CodeSignature/CodeResources" lists 5482 files that should be present in the folder containing the library but only 199 were found since the folder was cleaned up for unused files.
In other words, the lib signature is not up-to-date since the context has changed since the last time it was signed. Resigning it solves the case.
Not yet solved
takinmod_py use shared memory
The original code crashes with no specific error in the Console.
Here the suggestions of Sam Rowlands (App Wrapper 4).
When doing shared memory in a Sandboxed Application, you must use the App Sandbox safe functions for creating and accessing shared memory.
You have to use shm_open to create/open the memory.
You have to use mmap to create a local ptr.
For the identifier you must use an identifier that starts with the code signature's team identifier. The team identifier is the code in brackets of the code signing certificates name.
// --- The identifier is incredibly important and must take the form.
// teamID.appID/instanceID
// But it is limited to 31 bytes, not characters!
// Your TeamID is the one from your code signature.
const groupID = "QXAFMEPH6X.akda"
Dim identifier as string = leftB( groupID+"/shm" + UUIDCreate, 30 )
https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/shm_open.2.html
shm_open -- open a shared memory object - shm_open() is specified in the POSIX Realtime Extension
https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mmap.2.html
mmap -- map files or devices into memory
-------- Doc Apple ------
Consol error messages
forbidden-sandbox-reinit
Sandbox: takinmod_py(4969) deny(1) forbidden-sandbox-reinit
Embed executable within a sandboxed app must have only the following entitlements:
com.apple.security.app-sandbox
com.apple.security.inherit
Entitlements checking:
codesign -d --entitlements :- Takin.app/Contents/MacOS/takin_pol
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.app-sandbox</key>
<true/>
<key>com.apple.security.files.user-selected.read-write</key>
<true/>
<key>com.apple.security.print</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
</dict>
</plist>
This command shows that "inherit" entitlement have been replaced by the bundle entitlements, presumably due to the use of "codesign --deep" .
Apple states that the use of --deep must be avoided and that auxiliary binaries must be signed/sandboxed separately.
com.apple.private.tcc.manager.check-by-audit-token
attempted to call TCCAccessRequest for kTCCServiceAccessibility without the recommended com.apple.private.tcc.manager.check-by-audit-token entitlement
The word private implies it might be an entitlement only available for Apple internally. There is very little information about this entitlement on internet, even not exist in Apple's development document.
forbidden-sandbox-reinit
Failed to produce a full report for: takin_pol[4610].
Sandbox: takin_pol(4610) deny(1) forbidden-sandbox-reinit
Sandbox: takinmod_py(4969) deny(1) forbidden-sandbox-reinit
Embed executable within a sandboxed app must have only these entitlements
com.apple.security.app-sandbox
com.apple.security.inherit