in Learning

Designing a Personal Knowledgebase

Update: 09-07-14: Due to the interest this post has generated, I’ve put a sign up form at the bottom of the post for people who are interested in being part of a collaborative effort to build a PKB solution.ย 

As a graduate student, much of my time is spent acquiring knowledge from various sources. Indeed, this is a quintessential feature of academic work, which I enjoy very much.

Since so much knowledge is now digital, there is no shortage of material from which I can learn. On the contrary, I’m usually drowning in too much information. But that’s a discussion for another day. For me, right now, the major problem is that I lack an easy and effective system for capturing and recording my learning. My memory alone will not suffice. What I need is a personal knowledgebase, which I define as an external, integrated digital repository for the things I learn and the resources from which they come.

Many have tried to solve the problem I’m encountering now, and numerous digital solutions exist. Some of the most popular options include Evernote, Devonthink, and Voodoo Pad. Over the course of my graduate studies, I’ve tried many of these programs, but all have fallen short of what I really need, given my own workflow.

Therefore, I’m writing this post to sketch out what my ideal personal knowledgbase would look like. I hope that, by doing so, I might get some clarity on what I really need and how I might be able to craft my solution. And, I hope that, if you, dear readers, find my design interesting and valuable, we might work on coming up with a prototype.

Requirements of my Personal Knowledgebase

The features of my ideal PKB are the following:

  • Minimal Effort to Capture and Maintain
    • This is probably the most important feature of the entire thing. I want the process to be as seamless as possible. Input needs to be super easy. Whether it’s grabbing extracts from sources, or doing manual input (i.e. typing stuff), the process needs to be fast and easy. A good example of the ease I’m talking about would be that of nvALT, a simple plain text program. Working just from the keyboard, whenever a thought occurs to me, I can instantaneously open up a dialog box, write a few lines, then close out and return to my work. That’s how my head works, and so I need tools that work in the same way. In the same vein, once information gets into my system, I want to be able to manipulate it quickly and easily, either with a graphical interface (see below) or simple command line functions. And, of course, I want to access and use my PKB with similar speed and ease. There is no point in having a PKB if I don’t want to use it because of bloat.
  • All in one place
    • Right now, different aspects of my makeshift PKB are in different locations and applications. This is a messy situation. In my ideal PKB, everything would be in one place – the resources, the extracts, the pages, etc.
  • Non-proprietary (open source) file formats
    • I don’t want to use proprietary file formats mostly because I want to make sure my knowledgebase is future proof. I don’t want to rely on a particular piece of software to use my system. Moreover, by using non-proprietary and open source files, I can access and use my PKB across all platforms. So, for example, I would (and already do) use text files for my text input, since this format can read and accessed across all computers, and it is never going away.
  • Ability to accept various kinds of input (text, video, audio, etc.)
    • I mostly work with text, but increasingly my learning includes video and image files, especially in science. So, I want to integrate all these different source types seamlessly.
  • Linkability (is that even a word?)
    • One of the most powerful features of the internet is hyperlinking. In the same way that the web is a dense network of pages, I want my PKB to be richly connected. Wikipedia is a prime example of a knowledgebase with abundant hyperlinking.
  • Semi-automated input and organization
    • Gathering information is fairly easy. It is the extraction and organization of the information that takes time. It’s true, all good things take work, but if the amount of effort needed to create and maintain a PKB is too high, I’ll likely not use it much. So, I need my ideal PKB to have functions to automate extraction and some elements of organization. That’s why I say ‘semi-automated’. I know full automation is not feasible, since the imposition of meaning onto the raw information is something that I must do, not the computer. But there are a lot of things the computer can do, and so, I’d like to make it work for me as much as I can.
  • Searchable
    • The ability to search is one of the major advantages of a digital repository, as we all understand.
  • Multiple organization schemes
    • Organization is the difference between a masterpiece and a mess. You can have the best raw ingredients, but if you don’t impose structure on those components, the value of those components is minimal. Meaning is about organization and relationships. Thus, I need my PKB to have multiple modes of organization. A folder (directory) structure is what we’re accustomed to, but there are other ways of relating information, such as outlines and mind maps. I’d like to seamlessly move between different ways of viewing the same information.
  • Web-based (plus or minus)
    • Where will my PKB live? It could be on my computer. I’m OK with that. But no one else would be able to take advantage of my efforts. I’d like my knowledgebase to be accessible to other people, and the web allows that. Another thing is that, if I will be utilizing hyperlinks frequently, I need to keep my PKB in the same place, otherwise all the links in the system will break if I ever move anything. Thus, it would be wise to start building my PKB in a place where I know I will never move it. Hosting on the web would seem like a good option, or maybe a personal server.
  • Accessible and operable via Graphical (GUI) and Command-Line interfaces
    • I want to be able to interact with my PKB by graphically (using a mouse to move stuff) or via the command line, for more niche tasks. I find using a GUI to ‘drag and drop’ things to be ideal for imposing idiosyncratic organization (I’ll say more below).
  • Cross platform
    • We all have multiple devices now. For me, I work from my laptop and my iPad, and sometimes iPhone. I want to be able to access my PKB from all these. This is another reason a web-based PKB would be good, since all devices, Mac or PC, mobile or desktop, can access the web through a browser.

A Personal Wiki – The Closest Thing to my Ideal

Many of the features that I’ve outlined here are available through existing tools. The closest thing to my ideal PKB is a personal Wiki, built on a platform like MediaWiki or DokuWiki. The problem with these tools though is that they lack the organizational features I’m seeking, and they also don’t have strong automation. You can create a rich web of hyperlinked information, but it all has to be curated and organized by hand (lots of typing, lots of time). Wikipedia works well because the effort is crowdsourced. But I’m just one person, and so I can’t spend all day pecking away to create some record of my learning, otherwise I’ll spend all my time documenting, rather than actually using my PKB.

I’ve used Devonthink Pro Office (DTP) for my PKB for a while, and it is quite powerful and has many of the features I desire. Academics, particularly those in the humanities, have used DTP to good effect. DTP got me only so far though. I could extract information, form hyperlinks, and use ‘Smart Groups’ to automate organization, but beyond the sorting into those Smart Groups, I could not add any additional organization, like hierarchical outlining. It would just be a jumble of knowledge nuggets that I couldn’t do anything to. Moreover, DTP is a proprietary database with its own linking scheme. If I ever were to decide to not use DTP, so much of my efforts would be lost. I’d rather not rely on that.

My ideal PKB would be a hybrid of existing tools. I’d like the open and easy format of a Wiki, the smart automation of Devonthink, and the organizational power of an OmniOutliner or Tinderbox.

I’ve seen some very interesting homebrew PKBs that also approximate the features of my ideal PKB. Prominent among these is the researchr platform created by Stian Hรฅklev, a graduate student at the University of Toronto. Long before I was thinking about this topic, Stian was tinkering with his researchr Wiki, which aims to integrate all the different sources he reads. This prototype inspired me to pursue my own PKB solution. Several spin-offs on the researchr wiki also inspired me, such as this one from a former graduate student at Carnegie Mellon. Still, impressive as they are, these personal wikis require significant effort to create and maintain, which would be a deterrent for me. Nothing will be completely effortless, but I want to minimize the amount of extra energy I must dedicate to capturing and documenting my learning. Thus, in its current form, researchr is not the right tool for me. But I really like the brilliant vision of what it could be.

Alternative homebrew solutions include simply using the file system on your computer with additional support from tools like nvALT and Devonthink. Jason Heppler, a history PhD student, describes this setup in over at GradHacker. I like the simplicity of the file system, and now that Mac OSX incorporates tagging, systems like these make sense. But still, key components are missing for me.

Workflow-Inspired Design

What I’m doing right now

So what do I actually want? To answer that question, I need to take stock of my current workflow.

The basic processes and products in my academic workflow are the following:

Input

  • Learning (read/watch/listen) from resources
  • Engaging with resources: Filtering and annotation
  • Capturing (extracting the best bits from resources)
  • Processing: Adding metadata and organizing
  • Using

Output

  • Peer-reviewed publications
  • Teaching materials

To help visualize this, I’ve diagramed how I work.

My Academic Workflow

My Academic Workflow

Resources

My workflow begins with resources. These are the raw materials from which I learn. I gather information mostly from texts in the form of PDFs. Occassionally I’ll read a paper book, and increasingly, I’m watching videos on sites like Coursera. When I’m in a meeting or other face to face situation, I also jot notes using my iPad.

Where do the resources reside?

Right now, I keep most of my resources in my file system. Scientific articles I keep in Papers. All other documents I keep either in Devonthink or on my desktop. Videos I mostly watch on the web.

This situation is not ideal. I’d rather have all my resources in one place. The best case would be to house the resources in my PKB. But right now, I don’t do that.

Engaging with resources

Only a fraction of the information I’m exposed to is important to me. In the case of text (PDF), I’ll indicate that something is important and worth capturing with a highlight. A certain piece of information might also spur me to add my own thoughts or comments via an annotation.

If I do nothing else at this point, my highlights and comments are locked within the PDF. This greatly diminishes the value of this information. What I need to do is free them from the PDF prison. For that, I need to capture.

Capture

In my current scheme, I capture by extracting highlights using a custom Applescript. It takes each highlight from the PDF viewer Skim and sends them as individual Markdown-formatted text files to Devonthink Pro.

An individual extract looks something like this. I took this highlight from a recent paper in the journal Neuron.

 

A Typical PDF article

A Typical PDF article

Markdown-formatted extracts

The corresponding extract

Each extract has:

  • A paraphrase
  • The original quoted content
  • A hyperlink to the highlight in the PDF
  • My own comments
  • A formatted reference

This is all the information I would need when using this extract in a downstream application like writing a publication.

Processing

Once inside of my Devonthink database, I can add other relevant information such as tags.

Also, using the Smart Group function, I can replicate a single extract to multiple folders. So, for example, the above extract might be appropriately placed in Smart Groups called “Wnt”, “TRPV1”, “DRG Neurons” based on the specific internal contents, or in higher level groups like “Pain”, “Neurobiology”, or the like, using tags. This sorting based on pre-determined criteria is what I mean by ‘Semi-Automation’. Using Smart Group and tagging, I don’t have to duplicate information in multiple locations. The computer does that for me. This is a huge time saver, and in my ideal system, I want to maintain these features.

But this is where my processing stops though. Beyond grouping text files in folders, I can’t impose any more structure. What I’m left with is just a bunch of text files, whose relationship to each other I cannot express given the constraints of Devonthink.

So I’m at an impasse.

Future vision: What I want to do, but can’t do with my current tools

I’ll now describe what I would like to do with my ideal PKB.

My workflow would largely be the same all the way to the point of extraction.

Instead of using Devonthink, imagine that my PKB has the structure of a Wiki, composed of interlinked pages.

Highlights from a PDF source would be sent to a kind of holding place, a ‘global inbox’ (to steal a term from Devonthink), where they await my attention. Here, I will apply additional processing such as adding comments and tags.

Then, based on tags and content, the extracts would be distributed to myriad ‘topic pages’ based on pre-determined criteria. The distribution mechanism I’m thinking about is pretty much what a Smart Group does now. Based on search criteria, extracts that meet those criteria will be sent to all the appropriate topic pages. Importantly, a single extract can be represented on more than one topic page automatically. With a conventional wiki, since all the content is hand-written, information must be replicated across several topic pages if it relevant to more than one. This is tedious, both on the input side, but also during updating, since you to go all pages where some information is represented and amend it one by one. I want to avoid this.

Using the same extract as above, this is what it would like inside of a topic page entitled “Wnt Signaling”. Each topic page has its own inbox, which is where extracts sit until I place them into the body of the topic page.

 

Wnt Signaling Topic Page with a new extract in the Inbox

Wnt Signaling Topic Page with a new extract in the Inbox

To move an extract into the body, I simply drag and drop with my mouse. Applications such as OmniOutliner (which I used to make this mockup) already have this kind of functionality. On the web, Workflowy is an exemplar of a sophisticated outliner that allows for drag and drop of content.

With the extract moved into the right place

With the extract moved into the right place

Then, once the extract is in place, I can choose to expand it to see the additional information that I care about. Since I don’t need to see that information all the time, I can collapse and just see the paraphrase, which is usually all I’m interested in. The metadata are only really relevant when it comes time to use this Topic page for writing a publication.

See the additional details, but not all the time.

See the additional details, but not all the time.

For me, extracts are movable parts, or, as Daniel Wessel at Organizing Creativity calls them, Lego Blocks. Any extract can be moved within a topic page, and also between pages. At the end of the day, there is just one extract, but it is replicated among the numerous topic pages where it is relevant. Any change I make will be propagated throughout my PKB, unless otherwise specified (there are times when you don’t want to propagate a change).

An additional way that I might move extracts into certain locations in the topic page would be to have subheadings or subsections with their own ‘smart folder’ search criteria. So, I envision that a subheading would have its own search criteria based on content and tag information, and so new extracts would automatically move under the subheading, without having to be moved manually from the topic page’s inbox. If I knew that very specific kinds of information would go into certain places in my knowledgebase, then this method would be incredibly useful. In this way, I could build dynamic, ‘smart outlines’ that are updated as new information comes into my knowledgebase. Konrad Lawson at the blog Munnin describes a similar idea for a smart outline, from which I drew inspiration for this implementation here. This kind of page would be incredibly useful for writing publications, since as I’m reading and capturing, I could also be sending knowledge nuggets to one or many topic pages that would support my later writings.

Other features that might be interesting would be the ability to hover over an extract and see all the other pages where that extract resides. I could also see a feature that would visualize these lateral relationships using a kind of network graph. Getting a big picture view of all the connections would support creativity and idea generation.

The Payoff – How Would I Use my PKB?

So what do I get for my fiddling? Why do I even want this elaborate structure of information and how would it benefit me?

My number one use case would be when I’m wanting to write a research article or review, and I ask myself “What do I already know about X?”. Right now, I rely on my memory to direct my efforts. I can search my Papers database but that usually just takes me to whole articles, rather than to specific content within the paper. I don’t want to re-read an article. I want to go right to the meat. Without a good PKB, we in academic work reduplicate our efforts and waste time. Moreover, with the moveable extracts, I could drag and drop the best parts into an outline for the paper, and this would support my writing. No need to type everything out again if I’ve already invested that effort. Walton Jones describes how he uses his own custom PKB to support his learning and writing academic papers. My system would perform similarly for me.

Another important use case would be to use my PKB pages to support or template some kind of activity I want to do. For example, recently, I’ve been learning to use R to do statistical analysis of some biological data. Learning by doing is my favorite way to learn. As I’m going through the rigors of working up my data, I’m learning many useful tips and procedures, ones that I’m sure I’ll have use for in the future. I glean these nuggets from PDF books, website and videos, predominantly. When I return to doing the same kind of work in the future, I’m going to want to quickly locate the right resources, and to go directly to the relevant parts, and not have to wade through a whole book or website. Having an ‘R Analysis’ topic page with all my extracts, insights and comments would be incredibly useful. I’ve informally been keeping a kind of topic page in nvALT with some links or notes to myself regarding where the good info lies (e.g. see ‘Learning R page 14’) but this is not really scalable or sustainable. I need something more robust and systematic, such as the PKB I’ve described.

I would also like to use my PKB to share information with other people. Using my R example again, I know I’m not the only person in the world who would like to have a crib sheet full of R tips from around the web and from books. I’d love to send my PKB pages to colleagues who might also benefit. I’ve experienced this firsthand with the Learnstream wiki. I came across an informative page on concepts and it helped me write a paper I was working on. I’d love if other people could benefit from my research efforts in the same way.

Conclusion

So that’s my ideal system. A kind of hybrid between a Wiki, Devonthink and OmniOutliner. Given that there are existing tools that already perform certain of the functions I seek, I know what I’m proposing is in the realm of the possibility. But it’s now a matter of bringing it all together. The workflow I’ve described is my own, but I imagine that many other academics work in a similar manner and could benefit from such a tool.

I’ve described my ideal PKB from the user’s perspective, since I don’t have the coding knowledge to conceive of what the backend would look like. For the coders out there, what would be necessary to build something like this? Is it feasible to adapt existing Wikis and add an OmniOutliner-esque interface? Is it reasonable to want to store all my resources (PDFs, images, videos, etc.) on the web? Maybe with my own server? I’m revealing my ignorance here, but I’d love to know how I could craft the system I’ve described, and I’m not sure where to start.

Anyway, that’s what I’ve got. As always, I’d love to hear comments, criticism, feedback, etc.

For those who are curious to try out existing tools, I’ve compiled some resources below.


Resources

Off-the-Shelf Tools

Homebrew Solutions

Random Articles and Blogs on Capturing Knowledge


Want to help build an awesome Personal Knowledgebase?

Sign up here



  • Daniel

    Hoi,

    I’ve got a strange sense of deja-vu when reading your posting — because I have been there. Okay, I send another posting online prior to it’s scheduled due day: http://www.organizingcreativity.com/2014/09/updated-academic-workflow-2014/ . Anyway. It’s tempting to invest a lot of though into the perfect system. It’s something you can spend ages on fiddling with. Until it is truly perfect. But personally … screw perfection! What is it good for anyway? And for how long? Perfection needs an unchanging environment — unlikely in science. Focus on the products. What do you have to achieve (what DID those who came before you achieve?) — and where do you need support? Personally, I have found multiple ways to improve, e.g., Wikis (with JavaScript, you can automatize *a lot*: http://www.organizingcreativity.com/2009/09/ferret-frame-and-javascript-enhanced-wiki/ ), but at the end of the day, it’s not how smoothly your system runs, but what you produce with it. So focus on that. You cover a lot of things in your posting, including keeping data far, far away from that dreaded data island (see this PDF for other tips: http://www.organizingcreativity.com/oc2_1p.pdf ), but still, it’s a bit technology focused. The question is not what technology can do for you, the more relevant issue is: “I am doing my work, I stumble here and there, so how can I fix it?”

    If I had any recommendation to my former (PhD student) self I’d say: Stick to the basics. Make sure the stuff you get is somewhere you can find it again. Get the information you need with the source information. Keep the stuff flexible and tied to the source information. And for [whatever] sake, do backups and keep the old backups. Other than that … gain an understanding of the topic you are working on. That’s what you should strive for — not the perfect PKB. And the strange thing is — my workflow actually support’s it — okay, only if you add to it some high level thinking about how all the tiny bits of information play together. But it works for me, but that’s just me ๐Ÿ™‚

    Anyway, best of luck for your work ๐Ÿ™‚

    • Felix Hummel

      “Stick to the basics. Make sure the stuff you get is somewhere you can find it again.”
      Absolutely! Also been there, but nowadays “manual organization” just means writing stuff down and “use” means

      a) search it on the web (if you don’t remember) or
      b) search it on your blog

    • pnel

      Have you tried Together? http://reinventedsoftware.com/together/ I used it for a while, but switched back to NVAlt.

  • http://achamess.github.io/ Alexander Chamessian

    Daniel, thanks so much for the thorough response! I’m honored you took the time to read. I bet you have been there ๐Ÿ™‚ Your blog was what got me thinking about this kind of stuff at beginning of my PhD studies.

    This post was mostly for me to articulate what I’d really like to have. But I know I can’t wait around for it. What I do now is good enough – I read, I highlight, I keep some makeshift notes for myself. It gets the job done. It’s all too easy to become enamored of the system and to forget we’re seeking results, not a system. So I don’t let the perfect be the enemy of the good.

    But all the time that I’m doing this copious learning, there is nagging feeling that I’m not getting the most from it because I’m not capturing the good stuff. Is what I’m doing now sufficient? It is. But I think it could be better? Definitely. I’m not going to waste too much time searching for the perfect system, but if there are tweaks I can make to tools that are out there, I’d like to try ๐Ÿ™‚ Looking forward to reading your recent post on your academic workflow.

  • smargh

    To be honest, I haven’t fully investigated this yet, but Gitit by the same guy who makes Pandoc might be something worth looking into. Also, you could check out Caleb McDaniel’s gitit wiki to see something actually built with the tool. I want to sit down and play with it one day, but that day just isn’t today.

    As you probably know, I also think Evernote could do some real work for you. Especially on the wiki side. Also, it is super flexible with organization schemes and “extract” data types. These seem to be boons for you.

    Just some initial response thoughts. Great post tho. I really like the clear, well-thought out articulation of where you are and where you want to be. I could probably use such a thought experiment myself ๐Ÿ™‚

    • http://achamess.github.io/ Alexander Chamessian

      Thanks Stephen. I’ll check Gitit out, and I may need to give Evernote a second look. You deal with more text than I do. If it’s good enough for you, why shouldn’t it be for me? ๐Ÿ™‚

  • Frank Degenaar

    I found a link to your post in a recent tweet…

    Personal Knowledge Bases are just that… very personal. Often the nature of the material may dictate what platform one uses. I use multiple applications. One I’ve done a lot of tinkering with is Evernote, mainly since a shared public notebook is web-based.

    Here’s a post from my blog on how I built my own geeky database for finding any spoken word in an entire TV series:

    http://www.productivitymashup.com/blog/2014/8/23/geeky-database-evernote

    Also, I take advantage of Evernote’s OCR to index and find text within images. I have a database of 17,000+ comic strips in Evernote. The OCR works beautifully. Big plus… it was not necessary to type out the dialogue in the comic strips.

    The TV series database is shared on this site (with links to Evernote notebooks):

    anylineyouwant.blogspot.com

    The cool thing about WorkFlowy is that one can also share any parent node you wish, publicly too. It does a better job than Evernote, in the sense that you can create hierarchies. In Evernote it’s simply a note or a notebook you can share.

    One interesting idea I’ve been entertaining would be to share a WorkFlowy list/ node with multiple Evernote public notebook links, arranged however you wish. One can go to town on the organizational setup. You could represent (shared) notebooks within notebooks – or at least the links thereof in WorkFlowy. In so doing, one basically uses WorkFlowy to share a set of shared Evernote notebooks. That way you could, in essence, share an Evernote stack. You could even organize your entire ecosystem of Evernote Notebooks hierarchically this way, either including private links or not.

    Of course, we won’t be able to do a global search in WorkFlowy and Evernote… but I’m looking more at the sharing possibilities for personal databases… and also structuring things the way your brain might work, to get a better grasp of things.

    • Frank Degenaar

      Also… as an academic, you may want to take a look at Gingko app. It might be just up your alley for dissertations, etc.

      • http://achamess.github.io/ Alexander Chamessian

        This is great! I really like the modularity and the ability to write in Markdown. Thanks!

        • Frank Degenaar

          Haha, I tinkered quite a bit with Gingko to set up a task management system… don’t mean to spam you with links:

          http://www.productivitymashup.com/blog/2014/8/4/gingko-app

          • http://achamess.github.io/ Alexander Chamessian

            Thanks. No spam. That’s what this blog is for. I want people to point me to new things.

    • http://achamess.github.io/ Alexander Chamessian

      Thanks for the thorough response Frank. Agree that the personal knowledgebase needs to fit one’s unique workflow. But I think there are enough similarities among academic workflows that a common tool could be used. Capturing stuff is ubiquitous, and then organizing it is too. I’d say this latter feature is where one’s idiosyncratic ways show.

      Anyway… Re: Evernote. Do you not worry about stuff being trapped in there? Evernote uses a proprietary format, no? I know you can export as HTML though.

      I may have to give Evernote another look.

      I’d love to wrap the moveable, web-based outlining of Workflowy into a Wiki (mediawiki?) structure, so you have pages that are linked but that have moveable points internally, and that are replicated across all pages. It’s the same reason we use tags – some information can be categorized in multiple places.

      Anyway, thanks for the insights!

      • Frank Degenaar

        Not to sound like a crazed Evernote fanatic… well, I am!… but they are pretty phenomenal. I can’t, for example, get OCR in any other platform quite the way they do it. Also, Evernote is pretty phenomenal with their tagging possibilities. I don’t know of any other platform that exists that will give you the sheer amount of search options that are inherently part and parcel of Evernote. Of interest, you may enjoy a post on my blog, likening Evernote to the Human Intersect (from the geeky spy TV series, Chuck)

        What impressed me is Evernote’s ability to connect the dots in multiple contexts… for example, when you’re searching Google, there is an Evernote Web Clipper feature that allows you to do a “simultaneous search”. It will actually show you within your Google search results (to the right), similar content you already have archived in your Evernote account.

        Web clipping is another huge area. It seems like you might be able to use Evernote’s features more than most, judging from the multiple media sources you need to squirrel away. With the Evernote web clipper, you are able to directly clip web pages, PDFs, emails from your Gmail account, etc, directly into Evernote. There are many other ways Evernote has come up with to help people make connections, including “associated notes” within your account (premium feature). Anyways, in my post on “Chuck VS the Evernote” I expand on all of this.

        Another thought – you mentioned how you like to annotate your PDFs and send summaries to Devonthink through an Applescript. Evernote gives a summary of all PDF annotations (highlighted and otherwise) at the beginning of each PDF. That’s basically my “poor-man’s workaround” for the technical stuff you do. A lot of the technical terminology you use is a little beyond me… but what I do know for sure, is that Evernote seems to give some pretty easy solutions to many other things you have mentioned, without needing to wangle anything out of it, haha!

        There are even applications that exist that will do a consolidated search of your Evernote, Dropbox, Google drive, email etc. They would just need to include WorkFlowy and others to complete an arsenal of platforms that one might use to house their databases. It seems like there’s an easier solution than one might expect. The key is to fully exploit the strengths of each application and find a way to consolidate the “fractured” and disassociated data. It’s just a matter of time!

        • Frank Degenaar

          Also… nope… I don’t feel like my info will get trapped in Evernote. In fact, that is the very topic of a future post of mine, and I will draw on the Geeky references of 2001: A space Odyssey and another Star Trek Episode to illustrate how Evernote gives one complete control of your own information and the ability to extract it. They don’t hold you hostage. Their open API is highly adaptable, along with their ENEX (Evernote Export) file format.

      • http://plaeroma.com/ Will

        I’d like to second Frank’s comments on Evernote. As much as I’d prefer an open-source or self-hosted solution, nothing else comes close at the moment in terms of cross-platform usefulness. I do worry about lock-in, but my hope is Evernote will continue to offer ways out (HTML, e.g.) before anything is permanently stuck.

  • Pingback: Designing a Personal Knowledgebase | Infotentio...()

  • Pingback: Designing a Personal Knowledgebase | Linking Li...()

  • http://blainsmith.com/ Blain Smith

    Not sure if this is too simple for your needs, but I came across this the other day.

    http://scribbleton.com

    • http://achamess.github.io/ Alexander Chamessian

      Thanks! I saw this. Looks nice, but might not have the power I’m looking for. But I like the look and concept.

  • Khuram Malik

    I have a problem of a similar nature but different context. As a consultant I constantly need to learn about what’s new in my field, and collect resources. Most of the time, these are links. They’re rarely my own notes. I collect them because a) I need to be able go back and reference fast and b) often I need to share the knowledge with others. I’ve been trying a number of solutions from Evernote, through to Trello through to Kippt to name but a few. Even though I use these services extensively in other ways, they severely fall short as a PKB. Kippt was the closest thing there was and I was so happy and relieved to find it. There was no mobile app and pocket integration was promised but not delivered which would have made it feature complete as far as i am concerned. Sadly they abandoned development early this year and I’m back to square one.

  • kfogel

    A lot of us get most of this functionality from Emacs Org Mode (http://orgmode.org/). It requires a fair amount of up front investment before it becomes useful, and it’s not good for people who depend on visual layout (as opposed to symbolic/syntactical layout), so it’s certainly not The One True Perfect Solution you’ve been looking for. In fact, I might be more putting this comment here for others who read your interesting post and look at the comments than for you. However, if it sounds intriguing, give it a try! I’ve been happily embedded in Org Mode for a few years now, and can’t imagine going back.

  • Ryudo

    I suggest you trying Weavi (weavi.com), nonlinear text and its editor, which fits most of your requirements.

  • Sean Doig

    You’ve no idea how much this resonates with me. For about 2 years now I’ve been fleshing out much the same thoughts – almost exactly the same, in fact. I’ve assessed all the same products and reached all the same conclusions.

    I’ve started work coding out a few concepts in that time, but the requirement for automation kinda necessitates a decent level of integration with the OS which – for my skill set at least – complicates things quite a bit. Also having both granular semantic metadata and a decent user interface is *hard*, but when you start diluting these two features for the sake of practicality you end up back at a standard wiki pretty quickly.

    It’s a very tough problem space, but I remain convinced that a proper exploration of this idea could yield one hell of an app.

  • http://www.geyfman.net AnatolyG

    Check out one note. I really like the fact that unlike evernote it supports some level of note hierarchy (subpages below pages) and that you can write anywhere within the note, making it easy to organize text spatially.

  • http://blog.cybersnoopy.com/ CyberSnoopy

    I share lots of same thoughts of the PKB idea. I’ve been using mediawiki now for about 5 years. One of the reason is I used to edit a lot of wikipedia articles and feels ok with the system. The good part is the system has lots of plugins available (wikipedia’s fancy representation is a very good example), and you can develop your own with wiki Template. The downside is the syntax is a lit bit verbose and visual editing doesn’t work very well for advanced features. The other problem is to get media uploaded and linked is not very convenient, for this part, I use evernote as a backup.

  • http://michaelmusgrove.com/ Michael Musgrove

    Much of this marvelous tool you describe can be achieved with a self-hosted WordPress site, Platen (a Google Chrome desktop plugin,) and a few WP plugins and tweaks.

  • srinathh

    You may want to consider using a blogging focused static site generator if you want a completely open solution. If you use [Jekyll](http://jekyllrb.com/) or better still [Hugo](http://hugo.spf13.com/), you can simply write abstracts as markdown posts with links etc. Hugo in particular will let you easily use any hierarchy.

    • willurd

      I came here to say this, too. I have experience with Jekyll and I like it.

      It seems that, for existing solutions, a static site generator provides the most flexibility while also meeting most of your requirements (linkability, web-based, etc). Out of the box I would say static site generators don’t exactly support quick entry, but for that I’d try creating some simple workflows for Alfred (http://www.alfredapp.com/) like “inbox Some item text”, which could append an item ‘* Some item text’ to a pre-defined inbox file, or just “inbox” which could simply open your inbox file in your editor of choice.

  • Tobi

    Actively searching for the ideal personal knowledgebase has never led me to a solution I used for longer periods of time. I experimented with using a locally installed wiki, which to me proved to be a time sink. I would almost never go back to pages I had created.

    I also use the Emacs Org Mode, and to me it is a marvellous tool to help me be more organized. However, I do not tend to use it like a knowledgebase.

    The thing that has turned out to be the closest thing to a knowledgebase for me is the spaced repetition system Anki (http://ankisrs.net/). I started out capturing knowledge with the aim of memorizing it. I never had the concept of a knowledgebase in mind while doing that.

    However, I developed ways to structure knowledge in a way that made that knowledge more easily retrievable to my mind and, over time, made some customized note types to reflect some common knowledge structures like ordered lists, unordered lists or listing competing items that would often create confusion directly with the information they competed with. Also, over time, I started to use tagging consistently.

    These days, I do not only regularly review many things I entered into the Anki database, but I also use its searching facility. Searching for some tag or some word helps me to find specific items of information, and the tags also help me to rediscover the source of some disputed bit of information. Also, perusing the database helps me in attaining an overview of the limits of my knowledge in a given topic and gives me an idea where to continue.

    While reading your post and noticing that nobody has commented on Anki yet, I compared its features to your ideal criteria and saw that Anki fulfills many of them to some degree. Anki is not designed to be a note taking software for referencing notes, but more like a note taking software for repeating notes and thus memorizing them. However, as I said, for me it has unintentionally come to also serve the former purpose.

    I will try to go through your requirements item by item:

    Capture has become very quick once I got used to it and especially once I set up custom note types. Maintenance is easy because I tag information and I suspend information I do not want to repeat anymore. The card browser is quick in my database of about 40,000 cards (of which I regularly repeat about 10,000 at this time) and the browser makes use of tags, search items and further parameters nicely.

    Anki makes use of a database and stores media files in a dedicated directory, all in one place. It also features its own cloud, and its format is open. The author has started out developing it as open source and I think it has largely remained that way. Notes, however, are not always stored in pure text, but make use of HTML and CSS, so this deviates from your requirement. I also use Javascript on one note type for which I need information to be presented in a random order.

    I can and do input audio. Have never tried video, but this seems to be supported as well — basically, in Anki, you can do many of the things you can also do in a web page. There is LaTeX support, which I use.

    The cards are not designed to be interlinked, I think. From the manual, “Anki supports links between cards of a note, but not between unrelated cards.” This could be seen as a drawback, depending on how one intends to use the knowledgebase. Links to external sources are supported.

    In terms of automation, Anki can e.g. import a CSV table and make it into notes and cards. Also, some of my note types have become involved to the degree that I take one note, entering information into fields, and it automatically generates ten or so cards, and in a few instances even significantly more. Tagging provides a light-weight way to organize the information, however, it is not purely automatic.

    Search is well supported by the card browser. In terms of organization schemes, Anki is clearly designed with tagging in mind, however, it supports add-ons and exporting, which makes alternative ways to organize and present the information at least possible.

    The cards in Anki use web technologies and the templates use CSS designs as well as HTML features. Javascript is available to some degree. The data is optionally synced to the cloud and can be shared with others. In the context of Anki, this is clearly provided with teaching classes in mind, and the step from a personal to a collaborative knowledge base would not be much fun with Anki, I think. Data storage is always done using one local database and optionally a mirrored cloud based database, as well as a collection of media files. However, as mentioned above, internal hyperlinks are not natively supported.

    It is probably possible to design CLI tools for some purposes. However, Anki is GUI-based. Also, Anki is available on many platforms. I think it costs money for some mobile versions.

    I only did this comparison because your article made me notice that I came to use a software for the purpose of a personal knowledgebase that was clearly not primarily intended for it. Going through your requirements was a fun thing to do. I do not know if my list is helpful to you directly. It is clear that Anki does not provide everything in the way you have it in mind. Thank you, however, for an article that inspired some reflection about the way I started to develop a personal knowledgebase without noticing it ๐Ÿ™‚

  • http://jamesthornton.com James Thornton

    Working on a system close to what you described: http://ptrs.io

  • http://jdu-it.co.uk Jeff Uren

    Two other Off-The-Shelf apps that I’ve been using quite a bit that have some nice features:

    1) Ulysses III – http://www.ulyssesapp.com/ It’s entry is markdown, but you can export to HTML, PDF and other formats, it’s Mac Only but there’s an iOS app coming some time in the future. You could potentially set up a background service that watches a folder that you export to HTML to and have the service upload the folder to a web server, I use it with github to manage change control but using a manual process where I have a single branch in git that I push to whenever I create a new export, the trigger is using a custom workflow in Alfred on Mac after I export to a specific folder.

    2) Scrivener – http://www.literatureandlatte.com/scrivener.php It’s main focus is writers, but again, it can be manipulated to handle different type of content, you can create page templates and enter information into them and then link to them from your main documents within a project. The main use case is to have character sheets, etc… but I’ve used it for things like describing services in an architecture, components, etc… and creating linked references in the main documentation that I’m writing.

    Not necessarily outright solutions for you but they both have pros and cons that you can look at to see how some other apps have attempted notes, writing and organization within an application.

  • Oliver Burkill
  • CoLearnr

    Take a look at CoLearnr – http://www.colearnr.com (I’m the founder)

  • leobard

    Checkout nepomuk.kde.org and everything you can find on the idea of a “semantic desktop”.

    It is what you describe, but on a “what if Tim Berners-Lee would do it” level.

    If you nee infos, contact me (Leo Sauermann), you can find me on google.

  • http://www.elroyjetson.org/ James King

    Utilizing your PKB requirements, I sat down and thought about how best to make this work. This is a sketch of what I came up with that meets all of your requirments.

    ## PKB Requirements

    1. Minimal Effort to Capture and Maintain

    [Export Highlighted Text in PDF (Acrobat)](https://www.youtube.com/watch?v=JUnnLVBzlJ8)

    [Using Automator and Preview](http://www.macworld.com/article/2042365/automator-workflow-of-the-month-powerful-pdf-tricks.html)

    Markdown

    Safari Webarchives

    Bookmarklets – [Bookcision](http://www.norbauer.com/bookcision/) to get highlights from kindle books

    2. All in one place

    All in the filesystem on your machine, and portable with Dropbox

    3. Non-proprietary (open source) file formats

    – Markdown (plain text)

    – PDF

    – MP3

    – MP4 (h.264) or Youtube link

    – jpeg or png

    – webarchive (mostly portable – http://en.wikipedia.org/wiki/Webarchive)

    4. Ability to accept various kinds of input (text, video, audio, etc.)

    See item 3

    5. Linkability

    Store PKB in dropbox, a link can be generated using the Finder context menu

    6. Semi-automated input and organization

    Combination of using built in tools (e.g. Safari, Preview, etc.) and Automator scripts.

    Also utilize a files metadata cmd – i

    7. Searchable

    Spotlight

    8. Multiple organization schemes

    Combination of folder structure, Make Alias from Finder context menu, and Smart folders

    9. Web-based

    Not exactly web based, but since every object can have a url, you can use a Wiki, WordPress, Blogger etc.

    10. Accessible and operable via Graphical (GUI) and Command-Line interfaces

    Since this scheme only uses tools that ship with OS X (mostly) you can generally do both.

    Applications used:

    – Safari

    – Preview

    – Sublime Text 2

    – Finder

    – Automator

    11. Cross platform

    The files are cross platform. I don’t use Windows, but assume that you could find tools that are similar to accomplish the same thing in Windows and certainly on Linux.

    This works very well in OSX. Thanks for the article which got me to sit down finally and figure this out.

    • http://achamess.github.io/ Alexander Chamessian

      James – Thanks for taking the time to consider my requirements and to design your own workflow. Impressive!

  • John Tait

    +1 for org-mode. You can use it just for taking basic notes and learn about the many things it can do as you need them.

  • Roko

    How about using a dedicated gmail account?

  • Anthony

    Try TiddlyWiki http://tiddlywiki.com/

  • http://marcusvorwaller.com Marcus

    I’ve been using TheBrain (www.thebrain.com) for 5+ years to do most of what you mention. If you can ignore the name of the product and ignore their marketing and website, all of which are no bueno, and give their product a try, it may be worth it, if nothing else to give you another way of thinking about the problem.

    My “Brain” has almost 10,000 notes in it, all of which are instantly available via search and linkable to each other. The main feature is that any note can be a parent, child or “jump” from any other note, but that’s just the surface. Check out the screenshot I attached to get a better idea.

    The price is relatively high, the format is not completely open, but you can export it to XML and HTML and do what you want with it.

    If anyone’s interested in working on something better, marcus@vorwaller.net.

    Screenshot: https://www.evernote.com/shard/s1/sh/96c76b3a-630a-420a-b175-3f3689532943/d428404688c7bb9060451efe5d8ec323/deep/0/Marcus—TheBrain-8.png

  • davealbright

    Great conversation starter! I keep drives full of information in hopes of one day pulling all the file systems into a beautiful and intelligent database.

    DevonThink Pro will is an amazing company and dedicated. They will evolve into a solution similar to what you describe (no affiliation). I also installed the brain and was amazed but abandoned it due to the proprietary file sys.

    Each object needs to have a hierarchal convention to define the object’s characteristics and content.

    If I Search for something i want to expand beyond the query and go back to the browser session, objects viewed before and after. Basically, anything i can recall about the item is searchable and cross referenced (Tags, browser, time, time adjacent, location, general theme, kk, author, domain, project, etc.)

    • Pavel Doleฤek

      I’ve tried Devonthink and I like how they encourage advanced users… but something about the app doesn’t seem right to me… They try to be so powerful, but at the same time you cannot modify keyboard shortcuts, UI seems a lot old-school and just using the app doesn’t feel right to me… but after your comment i might give it another chance.

  • http://www.twitter.com/bennesvig Ben Nesvig

    I’ve been doing the same thing, creating my own book of knowledge. I read so many books and articles that I had to start organizing my thoughts and the best insights. I started with Evernote, but realized I’d have trouble exporting. I’ve since moved over to Scrivener, which is like Evernote, but allows me to export in any format.

  • Pingback: Designing a Personal Knowledgebase | Social med...()

  • Treycent

    Let’s say you have some knowledge captured in wiki articles and some in DropBox files and some in YouTube videos and some in SoundCloud audio tracks. One way to easily capture and link to all of these disparate knowledge sources is to use a site like Treycent (https://treycent.com).

    It lets you link spoken (or typed) questions and phrases to any URL even those behind password-protection or firewalls. Basically, a question-answering paradigm. Spoken input is supported on Chrome via the HTML5 WebSpeech API or any smartphone mobile browser.

    There’s no need to move data Treycent just needs a URL that you have access to and an associated question or phrase. For example, let’s say you wanted to capture the knowledge preserved in this wiki article about transfer pricing: http://en.wikipedia.org/wiki/Transfer_pricing

    You would copy the URL and tag it with a question like: “What is transfer pricing?”. Since, you can have multiple associations you could also tag the same URL with the phrase: “Explain transfer pricing”. You could then retrieve that knowledge in the future from Chrome or your smartphone by just speaking one of those phrases.

    You can also share “voicelinks” with anyone you wish by email. (Voicelinks are private by default.)

    The web site is free to use and I am seeking feedack – so please give it a try! I am also working on an Android Wear app so you if have a smartwatch and would like to be a beta tester let me know!

    Some videos to get you started can be found here: https://youtube.com/treycent

  • arthole

    The backup:

    Something you didn’t mention in your requirements which I find to be critical (and everyone does when your work disappears) is backups. I use Evernote for most of your list, but evernote SUCKS at publishing. It’s just really horrible for that. I do all my writing there, all my clipping, keep all kinds of documents in it. and they are available on all the computers I work on and on my phone.

    Evernote’s multi-platform and cloud availability and the multiple backups on different machines gives me a sense of security. But yeah, I’m writing my own wiki-fication software to get my stuff out in a publishable way.

  • http://blog.kellie.wildroseandbriar.com/ Kellie Miller

    Check out Dave Winer’s Fargo (http://fargo.io). It uses Dropbox for storage and is written in Javascript so it runs in a browser. I also use a lot of different tools for my PKB but Fargo is one piece of the solution for me.

  • Pavel Doleฤek

    I would love something like “Evernote Pro”
    * less fancy, minimalistic UI (Atom could be example)
    * markdown (Mou or MacDown could be example)
    * more shortcuts (any code editor could be example)
    * extensibility with Workflows/plugins (Alfred can be great example here)
    * advanced search (just anything is better)

  • http://www.nakedknowledge.com Naked Knowledge

    While my own needs are not exactly the same as yours, I feel your pain. It’s been years since I was in school drinking from the firehose of information but I never felt the existing products allowed me to properly capture the personal knowledge I was gaining. With the cost of higher education in the US, it makes you wish you’d invested in a proper PKB when ten years after graduating you can barely remember the most valuable things you know you learned. I’ve been working on a product called Naked Knowledge, which is the solution I wished I had when I was in school – or even now since the learning never ends. If you’re interested in following us you can sign up for the beta here: https://www.nakedknowledge.com. Cheers and good luck!

  • dbarr

    I use Dextrous ( http://excessivelyproductive.com/ ) as a sort of personal knowledgebase. It has a lot of the features you mentioned, but hyperlinking is unfortunately absent.

  • http://cogzest.com/ Luc Beaudoin

    Thanks for this, Alex. Part 3 of Cognitive Productivity deals with this issue. Also, it presents the concept of productive practice. There are also the concept of “meta-doc” (a file containing one’s comments on a resource). It presents solutions to the problem of accessing meta-docs. Advocates the “2 second” rule for accessing key resources (launchers like LaunchBar are a key part of this.) https://leanpub.com/cognitiveproductivity/

    • http://achamess.github.io/ Alexander Chamessian

      Thanks Luc. I’ve looked through your book before. You definitely have framed the problems well and offer good solutions, while also noting that the optimal tools aren’t here yet. Someday…

  • Jim Witherspoon

    Alex, thanks for your very interesting post. I posted a link to your post and it has spawned a discussion at http://www.outlinersoftware.com/topics/viewt/5521 . There is some discussion about DevonThink and ConnectedText – which may fit some of your requirements very well.

    • http://achamess.github.io/ Alexander Chamessian

      Hey Jim. Thanks! I’ll check out the thread.

  • Martien van Steenbergen
  • Paulo Eduardo Diniz

    Alex, great narrative of a huge problem and proposed solution! I too have similar feelings and judging by the amount of people interested, there is really a collective itch to scratch, to use your terms. I’d like to contribute with some thoughts. I love Notational Velocity (NV) and do think the so-denominated ‘non-modal’ functionality (that is, easily switch between searching old data and adding new data) is primordial for long-term building of knowledge, because on the long-term, you can easily forget if that tidbit of info is already on your “PKB”, and it also proves to be a nice tool to quickly acessing old excerpts. But more than that, i think that a heads-up-display dialog pane that works in a way similar to NV is also important for functions like reorganizing material, finding existing similar excerpts, pinpointing the original source of that excerpt, and also duplicating and further derivating pure textual data. That is a important issue to tackle, IMO. Other need I think people don’t realize but they DO HAVE is something like NV that works with more “delimited” Data, where each record has custom attributes (datafields) like a spreadsheet or database. I’d love to have “Notational Velocity for Spreadsheets edition”, and I also think there is no reason for it to be apart from an all-in-one solution for building a long-term ‘PKB’. As you said, it’s all about enpowering the user with multiple organization schemes, and such structured data is one of those very usual schemes in daily life, and involves basically pure data (text, numbers). Finally, i also reckon that web/cloud based is THE way to go when you’re talking about building long-term (presumably life long) knowledgebase, because you want it to be social (by that i mean sherearable when intended by the curator – e.g. in workspaces or in the academia) and you want to be accessible all the time from multiple devices. I also relate this concept of such ‘WEB-based PKB’ with projects like `Genius.com`, as one can notice from the reading of the exposition of motives on why Andreessen Horowitz VC-firm acquired the rapgenius.com site (http://news.genius.com/Marc-andreessen-why-andreessen-horowitz-is-investing-in-rap-genius-annotated). In sum, It’s all about building a layer of personal (or even collective) meaning to external information, processing this data, and using it to create new information. It’s the greatest challenge (the next step) of the information age, really, and the one to solve it, will make it big-time. Cheers. Paulo

  • Gene Bellinger

    I wrestled with the same dilemma for a long time until Diigo became the center of my universe…

    * https://www.youtube.com/watch?v=AHuDu1dmlsE

  • http://www.docear.org Joeran

    your idea sounds really nice. have you had a look at Docear http://www.docear.org? It features already parts of what you imagine, though not everything – in particular, the web storage is missing (andbut yes, it’s open source).

  • Pingback: Designing a Personal Knowledgebase | academic w...()

  • https://twitter.com/sahuguet Arnaud Sahuguet

    You should check https://www.zotero.org .
    There is a chrome extension to grab content for the web extracting the metadata automagically.
    They also have an API.

  • robinmessage

    Popcorn was an interesting research project trying to do some of this.

  • Treycent

    The diversity of “solutions” points to a need for tools that can interoperate seamlessly especially if you wish to share knowledge with team members.

  • Pingback: Designing a Personal Knowledgebase | Inteligenc...()

  • http://www.vipulschawathe.ind.in/ Vipul S. Chawathe

    Based on shortcomings of existing off-the-shelf stuff, collaboratively implementing another wiki-based CMS maybe an option. Some indie web folks are collaborating on much of these. The points made herein should be incorporated in the elective. Has participating in one of the web camps organized through https://www.indiewebcamp.org/ been done?

  • Dong Song

    I’ve been searching the solution for a week or so.I am a phd student and doing my thesis project.I ‘d like use a simulation analogy for the core function I need the best. It is also like a blackboard with memory. It can present items and their links you construct.you can simulate any combination of items with any linkage.I like to call it presentation engine,which presents simulations to yourself .

  • Scott

    Hey Alex,

    Great blog post. I too am an avid learner & have long wished for some of the features you describe in your Personal Knowledge Base.

    For me, the biggest requirement is a product that can help me retain the knowledge I’ve learnt (forgetting what I’ve learnt is so wasteful). Products like Evernote are great for storing information, but much of the information I capture I prefer to keep in my head – it’s much more valuable to be able to “know it when you need it”.

    After regularly expressing such thoughts to a like-minded learner, we decided to start building a product to scratch our itch. So, we’re building http://revisy.com

    We’re only just starting, but our goal is to use technology to help ourselves & other life-long learners learn more efficiently.

    One feature we particularly wanted was the ability to easily revise information using a spaced repetition technique, so that was the first thing we did. We also wanted the ability to easily import our Kindle highlights, so that’s done too. We’re contemplating lots of different directions, including integrating a web-clipper so we can easily capture information from all over the web, but, we want this product to be for all life-long learners, so we’re keen to hear ideas from other about ways of helping people learn more effectively. I’d love for you to try out http://revisy.com & tell me what you think.

  • Adam

    I’ve also been looking for a better solution, since it wasn’t until “too late” that I realized that Evernote is really just an information black hole, and not a useful tool for organizing and productively using that information. Since the middle of last year I’ve tried about a dozen tools. I gave up on perfection, now I’m looking to just to minimize the working tool set so I’m not wasting time or losing information in the “work flow”.

    I would love to participate in a working group to flesh out a grand scheme. And given that I program on the side for iOS/OSX, I could lend some skills there as well.

    I do think your “minimal effort” requirement needs some more thought. The reason why I say this is exactly why I left Evernote. There are about a thousand ways to get information into Evernote (different platforms, browser plugins, quicker note apps, reeder, twitter (seriously, 120 characters are worth memorializing?)). The problem is without some modicum of effort expended on the front end, all it leads to is hoarding and a giant pile of disorganized crap. There has to be some structure. And beyond that, you have to be specific about what is useful.

    For instance, many of us read hundreds/thousands of journal papers in our, and other, fields. However, while it will be nice to link extracts and notes to the paper, I sure as hell don’t want the body of the paper in my knowledge database. For one, I don’t need gigabytes of space wasted on lit reviews that cover the same topic and reference the same papers. I also don’t want to import any erroneous conclusions made by the authors. Which leads me to another point. There has to be the ability to remove information, or mark it outdated at the least. It might be nice to track the history of a topic, but I don’t want outdated information to be prominently placed.

    *addition* On the gigabytes of wasted space comment: Please note that I’m not talking about space as a constraint – we all have probably 8+ GB on our phones for crying out loud. The issue is USEFUL information. Using a knowledge database should not feel like searching for a needle in a haystack. Which is why it should be curated and not some dumping ground that you hope some sentient AI will figure out for you (hint: DevonThink’s AI and IBM’s Watson have decades to go).

  • diymanik

    Hey great article. I’m trying to build a learning website that’s essentially a collaborative KB at http://madrasa.ca. If you could take the time to try it out that would be awesome. Even the smallest bit of feedback from you would be greatly appreciated.

    • Andrea Barraza

      Hi, I found this post searching for a PKB solution. I find your idea of a learning website pretty interesting. Did you continue working on it? Have you thought of Recommender System functionality for it?

  • zxia31

    I agree that Emacs + Org mode is the most suitable as of now and arguably the following decades. After several years of using different note taking softwares or their combos, I finally turned to it.

    Since I am a CS graduate, my requirements are similar to yours but I also need code highlighting and LaTeX support that have ruled out many candidates. Although Emacs has a steep learning curve, once you grabbed it, you can write your own packages to fit your needs. There is a trade-off between proprietary (e.g., OneNote and Evernote) and open source softwares (e.g., most wiki-like clients). If it is proprietary, it may be more powerful and active (suppose it is still profitable) but less customizable. You can get all these benefits (powerful, active, customizable) from Emacs by paying the steep learning curve price. The Church of Emacs is full of gurus and followers.

    Another reason for using Emacs Org mode is that it is simple in the sense of form (plain text and interface). Since constructing PKB is a lifelong process, the KISS principle is extremely important. All the other PKB softwares based on GUI are likely to meet the bottleneck once they want to provide too many features, while most of these features are useless to you (Everyone only needs 5% functionalities but in the end GUI-based softwares become monsters.)

    As its name suggests, org mode is created for the organization, which is the essence of the PKB indeed.

  • Dmitriy Dotsenko

    My opinion on the best outliner:
    http://discoverfuture.blogspot.com/2012/09/best-knowledge-base-programs.html
    Anki could be used as the best knowledge base (including storing information
    and memorizing it) if it had outliner (tree structure) of notes,
    implemented in the main program or as a plugin.

  • Quang

    “Organization is the difference between a masterpiece and a mess.” – well put!

    I’m actually working on a new “Knowledge” extension for Evernote, that will give you “Folders” to organize your notes into folders. Which will give it needed structure.

    It’s dubbed, http://vanilafolder.com Launching soon, looking for early testers.

    Thanks for this great dissection on PKB’s.

  • Nik

    I’ve gone to the effort of making a website that handles ticketing, osTicket to be specific. With a built in Knowledge base it allows me to keep track of information that is asked of me and allows me a way to track any projects I’m on. This works perfect for me. Don’t know if this would appeal to others, but this does make for a very organized option.

  • KathlenSantana

    800fund.com obtain alternative lending solutions.

  • OK Michener

    Good to know. Now we know all pros and cons of the form. Try PDFfiller to fill IRS W-9 here http://goo.gl/Pk2AlR. It allows you to to fill out PDF files.

  • Heidi Hernandez

    Im attempting this with google drive/google docs, since google drive allows you to search inside documents texts and even text inside picture and is cross platform, and live on the cloud

  • Ghita Kassara

    Hi ! Great article. Very comprehensive. Why don’t you look at http://www.lifedigger.io, it is a website that fits some of your desires ! I would be glad to have your opinion on it.

    Basically, the idea is simple, it is a personal knowledge base that lets you build a durable knowledge. Think of it as a personal wiki, interconnected and intuitive, which also makes the user smarter. So you can easily go back to your notes and review them (even with the flashcard fashion …)

    Best

  • http://tevinzhang.com/ Tevin Zhang

    Hi Alex,

    Did you find the solution over the years? What’s the current state of your PKB solution?

    • http://achamess.github.io/ Alexander Chamessian
      • Supakorn Suttiruang

        Hi Alex,
        How is your current solution working out for you? I am currently building a new PKB-like piece of software, and it would be great if we can discuss its future.