This is a high level overview of how Datamaton’s Digital Manager works. This app (called Knit) lets you search, organize, backup and use all your digital content from one place. You don’t really need to know all this to use it. However, its nice to know a little more about an app you might use every day.
Knit is an indexer, just like Google & Bing search engines . . .
Similar to how Google and Bing index websites, Knit indexes your storage locations. That is, it makes an inventory of your content while leaving everything undisturbed in its original place. This process creates index files which can be queried to provide very fast search responses. Here are some examples of the kinds of information Knit saves in its index files:
- Emails: email sender, receiver, subject, date, size, message text, attachment info and email attributes.
- Text Messages: sender, receiver, message text, date, attachment info and the channel or group it belongs to.
- Appointments: start & end times, recurrence, subject, description, category, attachment info etc.
- Contacts: name, email address, phone numbers and attributes. Knit automatically harvests contact names and email addresses from email messages it indexes.
- All files: file and folder name, file extension, size, creation & last modification dates and file attributes.
- Photo, music and video files: album name, title, artist, genre, composer, play duration, camera, GPS information, thumbnails photo and other such standard information (e.g. EXIF data).
- Cloud files: user tags/comments and file sharing permissions, in addition to “normal” file properties.
- Compound files like ZIP, ISO and Microsoft Outlook PST files: Knit collects the compound file’s info, as well as info for files/emails embedded inside (as listed above).
Knit lets you search for your content using any of the attributes listed above. You might have a photo inside a ZIP file which is an attachment inside an email that’s been forwarded around a few times. Knit will let you search for the photo based on what you remember about it – e.g. an approximate date range when it was taken.
. . . But it doesn’t save anything on our servers
Unlike traditional search engines, Knit uses your computer to do the work. It saves the index files only on your computer. Even on your computer, it encrypts the index files using a password you provide, so only you can decrypt them. The index files typically take up less than 1% of the original content size they index. You have full control over what it indexes and how often.
A huge advantage of this approach is that nothing about your content ever touches our servers! We can’t use, sell or leak your personal information since we never had it in the first place! Searches are blazingly fast because they directly query index files stored on your computer. You can search when offline or when a removable device (e.g. USB/network drive, phone etc.) is not connected.
Knit is much more than just a search engine
Knit‘s objective is to “simplify your digital life”. Helping you find your content quickly is just one part of it. Knit is much more than an indexer. It is also an aggregator that lets you search, organize, manage and use all your digital content. There’s a powerful Knit capability that enables this – the ability to create arbitrary content selection rules. A content selection rule specifies:
- a) which storage locations it operates on
- b) which content type it operates on
- c) which content properties it operates on and
- d) how it tests the properties.
Here’s an example: From a) all storage locations b) select emails c) such that email date d) is within the range Jan 1, 2010 and Feb 1, 2010. Another example: From my a) C:\, D:\, Yahoo email account and Google Drive accounts b) select photos c) such that the “camera used” property d) contains the word “iphone”. You can combine together multiple such rules using “AND” and “OR” operators to create very powerful searches.
Here’s how Knit combines indexing and content selection rules to support different capabilities:
- Search: A search is just a combination of one or more content selection rules that may be combined with AND or OR operators. When you issue a search, Knit directly searches the index files it created on your hard drive – this is why Knit searches are so fast!
- Organize: Knit lets you assign tags to organize any type of content stored anywhere. In addition, it supports very powerful Virtual Folders that can contain any type of content, stored anywhere. A virtual folder is simply a collection of content selection rules, though you can also drag-drop specific content into it. For example, a virtual folder “Global Inbox” that contains emails and attachments in folder “Inbox” from all your email accounts can be made with just two rules: from all Data Sources, 1) select emails whose folder name contains “Inbox” and 2) select files that are embedded inside emails that are in the folder “Inbox”. These rules are dynamically evaluated, so new/future emails arriving in any Inbox are automatically included in this “Global Inbox” virtual folder. Since a virtual folder contains only rules and links to the actual content, it is very light-weight and flexible.
- Backup: Knit lets you move or copy any type of content from anywhere to anywhere. You can do this manually or set up automatic backup tasks. A backup task is actually just a collection of content selection rules paired with a backup destination and backup frequency. For example, you can create a task: “From C:\ select documents whose content contains the word “market” and copy them to Google Drive folder “marketing” at 10am every morning”. Knit runs each such backup task as separate background processes.
- Use: You can view all your content from within Knit and do normal day-to-day operations like read, reply or forward messages; move, copy, download, upload and share files, manage calendar appointments etc. This is obviously much more than what a search engine like Google or Bing would do.
Knit doesn’t force you to change what’s working well for you
You don’t have to stop using the apps and tools you currently use to benefit from Knit. For example: when Knit indexes your email accounts, it will not download messages locally and remove them from the server. It will not cause an unread email to be marked as read. Thus, you can continue to use your phone or browser to access your email. In fact, Knit will try to track the read/unread, forwarded & replied attributes of your email regardless of how you access it. So if you read and forward an email using your phone, Knit will also show this email as read and forwarded. Similarly, you can continue to use vendor specific apps from Slack, Google, Microsoft, Dropbox etc. and use Knit only to search, organize or backup that content.
Knit is a collection of applications that run in the background
Knit is actually a collection of many different processes and libraries.
Different components work together to do the overall job:
- Knit.exe is the user interface (UI) process and the program that you generally interact with. However, this program does not do the actual indexing, backups etc. Other background processes do that, so you can safely close the Knit user interface anytime without affecting anything.
- Cataloger.exe indexes and periodically re-indexes your storage locations in the background.
- Refresher.exe listens for dynamic insertion of CDs, USB drives, phones etc. so they can be indexed as needed. It also makes sure that periodic re-indexing and backup tasks run when needed.
- FileMonitor.exe tracks changes to local hard drives so that its index files get updated as soon as local content is created or modified.
- ImapMonitor.exe tracks your email accounts so new email is immediately indexed.
- TaskManager.exe runs your backup tasks in the background.
- SpaceMgr.exe periodically runs in the background to delete temporary or cached files created by Knit.
- DMInstallMgr.exe manages license changes and Knit installation and updates.
- Recrypt.exe lets you set and change the passwords used to encrypt your index files.
Knit starts and schedules these background processes automatically when needed. You need to interact directly only with the Knit.exe user interface.