Basically a DAM-System (Digital Asset Management). Includes a Webserver providing a SPA (Single Page Application) to browse the storage content.
Note
Work in progress, things might change and be not downward compatible
DAM: Manage Pictures, Documents...
Portable: Runs on Linux, Windows, Mac. No installation required, just copy. Can be copied to a external HD or Stick which then can be used to store assets from different devices.
Server included: Can be used on a NAS for example.
Deduplication: Same file is stored automatically only once, no matter how often it was added.
Resilience: Files are stored in directories, one per time period, to enable quick backups even on large storages. Meta-Data stored separately in JSON-Format. Database can be recreated from Meta-Data and vice versa.
Store files into an archive directory
- Deduplicated (same file only once, determined by content-hash).
- Organized in subdirectories, one per time period. Old directories will no be touched again to enable incremental backups.
- Meta-data is stored separately and updated when new files are added or existing files with different origin/owner.
- A database is created to be able to find/browse data.
- HTTP-Server included for Web&REST-Service.
- SFTP/SCP/RSYNC-Server included to receive files from remote devices.
To add new files to the archive
add [-skip-meta] [-check-hash] [-gzip] [-maxmem <bytes>] [-base <directory>] [-name <file-name-pattern>] [-r] <file or directory>
A HTTP Server which provides a Single-Page-Application to browse the storage
Note
This is work in progress, important features like TLS is missing at the moment
spa-server [-gzip] [-xor <key>] [-base <directory>] [-spa <http-root-directory of spa-app>] [-listen <ip:port>]
A HTTP Server which provides a REST-API to access the storage. This server is included in spa-server.
Note
This is work in progress, important features like TLS is missing at the moment
rest-server [-gzip] [-xor <key>] [-base <directory>] [-listen <ip:port>]
Update meta-data-database by reading all meta-data files and writing contents to database.
Not required if database is intact, because add also updates the database.
metadata-db-create [-base <directory>]
Accept files from remote computers via SFTP, SCP or RSYNC
Note: This is work in progress, important features at the moment
ssh-server [-host-key <files>] [-listen <ip:port>]
Add users, set/update passwords. These users will have access to Servers (Web, SFTP, SCP, RSYNC)
user-edit [-a] -u username
Use -a flag to add a new user.
Commonly used commandline arguments for asset-storage apps:
| Parameter | Description |
|---|---|
| base <dir> | Asset-storage base dir containing all data (file, meta-data, database). Default is $HOME/asset-storage |
| gzip | Use gzip to compress data. Important: Cannot be mixed, use always or never for one storage. |
| maxmem <bytes> | Max size in bytes when reading files while adding to storage. If a file is larger, it will not be read into memory and a temp-file will be used |
| name <patten> | Filter files matching file-name-pattern (*.jpeg for example) |
| skip-meta | When adding files: Skip updating meta-data if file exists. |
| check-hash | When adding files: Check content hash before adding. Faster only if most of the files already exists as it only calculates the hash in memory. Slower if most of the files are new because file will be read twice. |
| spa <dir> | HttpRoot-Directory which contains the SPA-files (HTML, JS, etc) |
| xor <key> | Content will be XOR'ed to obfusicate. This is to avoid manual changes to files (when content is XOR'ed, files cannot be openend and modified directly from storage directory) Important: Cannot be mixed, use always with same key or never for one storage. Important: Use same key for all apps with same storage directory |
| listen <ip:port> | IP and Port to listen to (Web-Server, Ssh-Server). |
I have used these libraries (many thanks to the authors)
- SQLite: https://pkg.go.dev/modernc.org/sqlite
- Web Services: https://pkg.go.dev/github.com/gin-gonic/gin
- Webp decoding: https://github.com/HugoSmits86/nativewebp
- Image scaling: https://pkg.go.dev/golang.org/x/image/draw
- MIME type detection: https://pkg.go.dev/github.com/gabriel-vasile/mimetype
- Web UI: https://vuejs.org/
- CSS: https://getbootstrap.com/