Hello, I'm Joseph Turner. This talk is about HyperDrive.el, peer-to-peer file system in Emacs.
HyperDrive is a JavaScript library developed by the HolePunch team for sharing files on a peer-to-peer network.
It's released under the Apache 2.0 license. Here's how it works.
To participate as a peer, you run a node, a lightweight local server that allows you to connect with other nodes on the network.
You can create a HyperDrive or multiple HyperDrives, and you can author files within them.
Each HyperDrive is automatically assigned a globally unique link that starts with hyper colon slash slash.
When you share that link with someone, they have access to your HyperDrive.
Anyone who has that link can load the HyperDrive from the network and view its content.
When you load a HyperDrive file from the network, your node caches that data locally and automatically begins seeding it back to the network,
making it available for others to download from you.
HyperDrives are single writer. This means that when you create a new drive, you are the only one who can make changes to it.
Others can view it and can seed it back to the network, but you're the only one who can modify it.
HyperDrives are offline first. This means that when you load data from the network,
it's stored locally on your machine for you to view later, even when you're disconnected from other peers.
You can also create new drives and modify your drives when you're offline and then share those changes once you connect with peers later.
HyperDrives are local first. This means that when you are connected with other peers on a local area network,
even if none of the peers involved are connected to the broader Internet, you can still share files.
HyperDrives are sparsely replicated. This means that you can download individual files from a HyperDrive without having to download the whole thing.
This saves on disk space and also allows you to quickly load just the files that you're interested in.
HyperDrives are mutable. You can add files, change files, remove files from a HyperDrive.
And when peers load your drive using the very same link, they will be able to load the latest changes that you've published.
HyperDrives are versioned. This means that when you make changes to a file, the previous versions of those files are not lost.
Peers can load the old versions of a file that was changed or deleted, for example,
simply by specifying the version number of the HyperDrive when the file still existed or existed in a previous state.
I'm presenting this talk on behalf of Ushen.
Ushen is a tiny nonprofit whose mission is to promote personal, community and global health through free and open universal shared information for everybody.
Ushen was founded in the early 90s by Paula Moss, Steve Nash and others with the goal of creating a distributed health information network that would allow people to find, share, compare and deliberate health information from a variety of sources.
Since then, the scope of the project has broadened to include all kinds of information.
And in recent years, we've been focusing on building with peer-to-peer software.
About three years ago, we started working with Mauve Signweaver, who has since then been our steadfast peer-to-peer explorer and guide.
This year, we started working on this HyperDrive.el Emacs package, and Adam has been the powerhouse behind the Emacs Lisp development.
About a month ago, Jonas Bernoulli started joining with us, and he has been offering his expertise in the realm of user interface design using his Transient library.
And Protesilaos Stavrou has been not only valuable in terms of user design and feedback, but he created a wonderful basic introduction to HyperDrive. Take a look.
Hello everyone, my name is Protesilaos, also known as Prod.
In this video, I want to show you the basics of HyperDrive.el.
This is a package for Emacs that lets us connect to the HyperDrive peer-to-peer network.
We can browse existing drives on the network, meaning existing file systems, or create and maintain our own HyperDrive to which we can add, remove or edit files.
We will see this together.
What I have here in front of me is a basic use package declaration for HyperDrive.el.
All I am doing is binding the HyperDrive menu to a key and also activating the menu bar entry of HyperDrive.
Let me show you very quickly the HyperDrive menu.
And this is the sort of interface that it brings up.
We will take a closer look at it.
As for the menu bar, we have the HyperDrive menu over here, and we will take a look at this as well.
Suffice to say that you can get the same functionality with the menu bar or with the HyperDrive menu, as well as by calling the commands directly with Emacs or their respective key binding.
I won't cover everything in that regard, but please bear this fact in mind.
Let's start then with what I have here in this other tab, which is a set of files.
I have prepared in my local file system a HyperDrive.
This contains a set of files that I ultimately want to share on the peer-to-peer network, meaning that I want this to eventually be reflected in my own HyperDrive.
What I did to get started is I invoked HyperDrive menu.
The very first thing you need to do to get started with HyperDrive, either to browse or to create, is to start the gateway, meaning to be able to connect to the peer-to-peer network.
You see here in the HyperDrive menu that there is an indicator next to the gateway heading telling me that the gateway is on.
I started it with capital G and then lowercase s.
Once you start it, you can type H to visit an existing drive, capital N to create a drive, or capital L to open a link to an existing HyperDrive.
You may get this link via email, for example.
In this case, let me be over here and let me just kill this buffer.
What I want in this case is to select a HyperDrive.
You see here I have my own HyperDrive and also the HyperDrive of Ashen, the developers of HyperDrive.el.
What I want to do in this case, you see there are various options available.
I want to type V to view a file.
And in this case, I will just type the forward slash, which means to view the root directory of this HyperDrive.
And I am here.
It is an empty drive apart from my credentials over here.
There is nothing more really to see.
But what I want to do is to start adding files to this drive.
Let me use the menu bar for this.
I will hover over to the menu bar and I will find the HyperDrive submenu.
Then I will go find where the drives are.
And you see that there are writable drives, meaning drives that I manage, and read-only drives, meaning drives of other users.
In this case, Ashen is read-only, of course, and Protesilaos is writable.
So what I want to do is upload a file to Protesilaos.
And it prompts me in the menu bar for a file.
And I will just add the readme.
And it is asking me, where do you want to add it?
And in square brackets, the default is to add it to the root directory of the HyperDrive.
So I will just type enter to select the default.
And there it is.
It is already in the drive.
Let me add the copying file as well, using the same method.
And I will go find my drive.
Upload file.
And let's add the copying.
And I also want to add it to the root directory.
So already I have two files.
This is looking good.
Now I am opening the file inside of HyperDrive.
This is not the same file as I have in my local directory.
And what I can do here, for example, is I can...
Let me open another file here, the copying.
I can create an org link.
Notice that these are org files.
So I will use the standard org store link command.
You can see the key binding I invoked and the command it calls on the top right corner of my screen.
And so what I did is I copied a link to this HyperDrive file.
And in this other file over here, I want to say free Libre.
So I am editing my HyperDrive terms.
And I will paste the link here for how to use my files.
And if we check what is here, you will see that this is a link inside of the HyperDrive.
And it's pointing specifically to that heading over there, which has this unique identifier, as you can see over here.
So this is wonderful.
And I want to rename it to check the copying for how to use my files.
And I will save this.
And now what I want to do is I want to invoke the HyperDrive menu.
And you will see now the menu, unlike earlier, has more commands, more stuff we can do with it.
For example, I can type W to copy a URL.
And by typing W, you will notice...
Let me go to the Scratch buffer to paste this in.
You will notice what the URL is.
It is my HyperDrive, my unique identifier.
And then forward slash, meaning the root directory.
And then readme.org.
And that is the file I was editing.
Let's do the HyperDrive menu again.
Let's remove the Scratch buffer.
And let's do the HyperDrive menu.
And you will notice that there are options to download the file, for example.
If you are reading somebody else's file, you can download it to your own file system.
Let me do that. Download.
And it is asking me, where do you want to save this?
For now, I will save it in the TMP, like this.
TMP, let's call it test.org.
Okay, tmptest.org.
Let me go and visit test.org.
And there it is.
It downloaded it just like that.
And this is how you can, for example, download the pictures and videos that I will eventually share on my HyperDrive.
Let's invoke the HyperDrive menu again.
And let's go up to the parent, you see, with the current sign.
And this will take me to the parent directory, in this case, the root directory of my HyperDrive.
Let me do it a bit differently.
The same idea, a bit differently.
For those of you who are familiar with dred and the dredjump command, dred is the standard file manager of Emacs.
And dredjump is a command that lets you jump from the current file to the directory that contains that file.
And so you see, I am here.
The dredjump command, by default, is bound to Ctrl-X, Ctrl-J.
So if I do Ctrl-X, Ctrl-J, in this case, it invokes a command.
You can see the name of it, hyperdriveup, which is functionally equivalent to dredjump.
It does the same thing, meaning that it took me to the parent directory of this file.
And I think this is very helpful.
This listing over here, in general, tries to mimic or to reuse the knowledge you already have of dred.
For example, if you type the letter O, it will open the file at point in the other window, same as in dred.
Whereas if you type Enter, it would open it in the current window.
Again, same as what you will do in dred.
Let's see over here.
You have options to jump with J, which is using minibuffer completion to go to a file.
Right now, I only have two files, but the idea is the same.
It's, again, what you would do in dred if you type J with the default key bindings, though, not with evil mode or something else.
Let's see again what we have over here.
You can create a bookmark, and this will work, but no need to show you everything.
The idea is that you create a bookmark the way you create any Emacs bookmark, to a file, to a directory.
It doesn't matter.
And then you can jump to it the way bookmarks in Emacs always work.
What I want to show you now a little bit is the history.
History in HyperDrive has to do with the drive itself.
Meaning that individual files do not have their own history, but the drive as such has a history.
Whenever you add a file, you remove a file, or you edit a file, you are incrementing the versioning of the HyperDrive by one.
So each action corresponds to one unit of history.
If you add a file, remove a file, and edit a file, this means that you are up three versions.
So whatever your version number is, plus three.
I am on version 24 over here.
Let me go to this file now, and let me do capital V and H to see a history of it.
You will notice that between versions 23 and 24, this file was constant.
But version 25, we have a change.
When you are in this buffer over here, you can type the equals sign, which is a key binding that will bring up the diff.
So the set of changes between the previous version and the current version.
And you see here between version 23 and 25, I have this addition.
And I think this is wonderful, because now you can always go and check what is the state of this file.
What is this person up to? What have they been changing?
And with HyperDrive, you can also visit, so let me go to the parent again.
Oops, sorry, what did I do?
Let me go to the parent here.
You can always visit a previous history.
So for example, I will go to the previous history, and let me see this file again in its previous history.
I am looking at the file, but notice that the file now is not editable, because this is in the past.
I cannot rewrite history.
I can only go to the present and then modify it and then create a new history, a new version.
But this helps me see the state of the file at that version of the HyperDrive.
So this is the basic idea of it, folks.
What I want to do then is continue with my process here.
Let me actually do it like this so that you can see.
Continue with the process of publishing all those files that I have on my HyperDrive.
For example, this is a picture of a flower that I have taken.
I think it's very nice.
And this is a video of an eagle that was flying above me.
And I will share this on the HyperDrive network.
And from the network, by the way, you can also stream video as well.
It is described in the HyperDrive manual, but I cannot show you everything right now.
I think you get the idea.
The gist is, you have a file system that you can share with the world using peer-to-peer technology.
And for me, this is a powerful tool.
This is a valuable proposition because I can share these large files I have, these pictures or videos,
which I cannot post on my website due to bandwidth considerations.
And this way, I can still share with the world something that I consider interesting.
That's all for today, folks. Thank you very much for your attention.
Remember that HyperDrive.el is still in development and things may change.
But the fundamentals are in place and will remain constant.
That's all for today. Take care. Goodbye.
Thank you, Prat.
Another fundamental feature of HyperDrive.el is drive creation.
The first step, as always, is to make sure that the gateway is running.
So I'll open up HyperDrive menu with Ctrl-C-H.
By the way, my key presses, as well as the commands that they run, can be seen at the top right of my screen.
Down here, I see that the gateway is off. So I'll start it with Shift-G-S.
Now, it takes a few moments for the gateway to spin up.
So to refresh the menu, I will close it and open it again.
And now we see that the gateway is on.
So I'll press Shift-N to create a new drive.
Now it's prompting me for a new HyperDrive seed.
A seed is a string of characters that will be used to generate, in tandem with my secret master key,
a new public key that will globally, uniquely identify this drive.
So I'll type in EmacsComp.
And after a moment, we see the newly created empty drive.
So I'll open up HyperDrive menu once more.
And I'll press H to open the sub-menu that shows more information about this HyperDrive,
as well as commands related to this drive.
In blue here, we see the seed that I just entered, as well as the public key that it generated.
We also see that the pet name is also set to EmacsComp.
The pet name is different from the seed.
The pet name is my personal, local identifier for this drive.
I can change it whenever I want.
And while it's not a secret, it's not displayed to other users.
So I will leave it for now.
EmacsComp is fine.
But when I go to share this drive, I'll want to display something more memorable than this long public key.
And that's what the nickname is for.
So I'll change that by pressing N, and I'll type in EmacsConference.
Now, when other peers load this HyperDrive by its URL, which I can copy by pressing W,
they will see the nickname in addition to the public key.
So if HyperDrive.el is like a phone book, the public keys are akin to phone numbers,
the nickname is like the name that your contacts give you when they introduce themselves,
and the pet name is the name that you actually write down in your phone book.
I'll show you what nicknames and pet names look like for drives that are not writable to me.
So I'll press Control-G to close this submenu, and now I'm back at the main menu.
I'll press Control-U-H to choose a HyperDrive, and I'll look at Prot's HyperDrive here.
And here we see that the nickname of Prot's HyperDrive is Protesilaos.
Now, it's grayed out, which means that I can't change it.
I can't change it because it's not my HyperDrive.
But I can change the pet name if I want it to show up under a different name.
So I'll press P, and I'll type in Prot, and hit Enter.
Now I'll open his HyperDrive by pressing F.
And I'll pick a path, I'll just hit Return to open the root directory,
And now, when Prot's HyperDrive shows up, at the top of the screen I see that it's identified with the pet name Prot.
Now I'll show off HyperDrive Mirror.
HyperDrive Mirror is like HyperDrive Upload File, except that it allows you to upload an entire directory full of files recursively.
For this example, I will upload the contents of the emacsconf-mirror directory.
First step is to open the menu.
I'll press H, and then I will choose the emacsconf drive.
In the bottom here, we see the Mirror group.
The first option that I can change is the source directory.
The source directory is the directory on my local machine from which files will be uploaded.
By default, the source directory is set to the current directory of the main buffer.
This is good for now, so I'll leave it.
The target directory is the directory in the HyperDrive where the files will end up.
By default, it's the root directory, but for this example, I'll put these files in a subdirectory called Notes.
The filter is the rule that allows you to programmatically determine which files in the local directory will be uploaded into the HyperDrive, and which ones won't.
By default, all files are mirrored, but in this case, let's say that I want to upload only the org files, these first three, and I want to exclude the markdown file, solarovennotes.md.
So I'll press MF, and I will choose the regular expression string option.
If I wanted to, I could choose a named function or a lambda, but I won't demo that here.
I'll type in org$, which will match against the files that end with org.
The filter is here, and I'll leave confirmation on.
The confirmation step just allows me to review the list of files that are going to be uploaded into the drive before it happens.
So I'll press MM to mirror them, and I see here that these three files are going to be uploaded.
Looks good. I'll press CTRL-C, CTRL-C to confirm the mirror.
Now it says three files have been uploaded, and here they are in the drive.
Good, so now I will modify the fermented overnight oats file in the HyperDrive.
So I've loaded it, and I'll add here four other grains.
Cook them in advance if you want to, and I'll save it.
Now this file, fermentedovernightoats.org, has been modified on the HyperDrive more recently than on the file system.
The file system file has not been modified.
So I'll go back to my local directory, and I'll modify a different file.
In this case, I'll add another hodown to the fiddletunes.org file.
Tom and Jerry. It's a good hodown.
And now I'll mirror again.
So I'll open the menu, and I'll press H, and then I'll open up the EmacsConf demo drive.
And now I've also decided that I want to include the solarovenoats.md file.
So I'll remove the filter so that it's no longer excluded.
I'll leave the rest of the settings the same, and I'll press MM again.
And now we see that the HyperDrive mirror buffer looks different.
So, there are two main groups. These are the files that are going to be uploaded, and these are the files that are ignored.
They're not going to be uploaded.
The first subgroup is the files that are new locally.
So the solarovenoats.md file is new on my machine, and it doesn't exist in the HyperDrive.
So the mirror command is going to take that file and add it to the HyperDrive.
This group contains the files that are newer locally.
So the fiddletunes.org file has been modified on my local machine, but it hasn't been modified on the HyperDrive.
So HyperDrive mirror is going to take the updated version and put it on the HyperDrive.
Now the first group that's going to be ignored are the files that are older locally.
So the fermentedoats.md file has been modified on the HyperDrive more recently than on my local file system.
So HyperDrive mirror isn't going to overwrite the version of the file in my HyperDrive with the older local version.
And finally the emacscomppreparation.org file hasn't been modified on either the HyperDrive or my local file system,
and the timestamp is identical.
So HyperDrive mirror is going to ignore this file as well.
So the only two files that are going to be uploaded now are the solarovenoats.md file and the fiddletunes.org file.
I'll confirm that with Ctrl-C, Ctrl-C.
And now in my HyperDrive we see that there are four files.
The solarovenoats.md file has been uploaded, and if I open the fiddletunes.org file,
we see that it now contains the line Tom and Jerry, which means that it was updated based on the change to the file on my local file system.
This HyperDrive mirror command is the command that we use to periodically update the Ushen HyperDrive with the contents of the Ushen website.
There's going to be a link to the Ushen HyperDrive as well as the website at the end of the video.
Now I'll go into a little more detail about the HyperDrive history buffer by showing off the readme file in Protts HyperDrive.
I'll press Ctrl-C-H to open the menu, Ctrl-U-H to be prompted for a drive.
I'll select Protts drive, then I'll press F to jump to a file inside of Protts drive, and then return to go to the root directory.
From here I'll press J to jump to an item in his directory.
I'll press return on the readme to load it.
And then finally I'll open up Ctrl-C-H to look at the menu.
Here I see that I'm looking at the latest version of Protts readme.org file.
I also see that the previous version of readme.org is unknown.
It's unknown because HyperDrives are sparsely replicated.
That means that when my node loaded this readme.org file, it didn't bother to load anything else.
It didn't load the previous history or any other file in his drive.
But now that I want to check out whether there is a previous version, I'll press Shift-V-P.
And when it loads, I see in the mode line down here that I'm now looking at version 25 of this drive.
That means that I'm looking at the readme.org file at version 25 of this drive.
I'll open the menu again, and I see that the same version number is displayed here.
The previous version, before version 25, is also unknown because we haven't bothered to load anything before version 25.
I also see that the next version, after version 25, is the latest version.
So I'll open that up by pressing Shift-V-N.
And now we are back where we started, at the latest version.
But now we see that the previous version, the version before the latest version, is version 25.
Because now that our node has loaded the previous version, it can display that information to us.
From here, I'll open up the history buffer. I'll press Shift-V-H.
And I can see that there are two known existent ranges in Prot's readme.org history.
This means that the latest version of readme.org was modified at version 39.
And that Prot made four changes to other files in his hyperdrive besides the readme.org file since then.
Before that, the time that Prot modified the readme.org file was at version 25.
And then he made 13 other changes to other files inside of this drive.
Before that, we don't know. We haven't loaded the history.
But since we're curious, I'll hit Return on the unknown line.
And now we see that in Prot's hyperdrive, the readme.org file didn't exist for the first 22 revisions of his drive.
Then Prot created it at version 23.
Then again at 25, made a change.
And then made another change at 39.
For good measure, I'll show you the diffs.
The first diff just contains the entire file because the file didn't exist before version 23.
Then at version 25, Prot added a link to the copying.org file.
And then at 39, Prot changed the link to his own hyperdrive to be a relative link.
Now, I'll stream a video from the Ushen hyperdrive that shows off a prototype interface we created for exploring networks of sources of information.
I'll open up Hyperdrive menu.
Press Ctrl-U-H to be prompted for a drive.
I'll select the Ushen drive.
Press F to jump to a file in it.
And then I'll jump to the media directory.
Once it loads, I'll press Return on the video that I want to stream.
This is a demonstration of the subjective trust interface that the Ushen team built for the u4u.io web app.
There it is.
Streaming a video from Hyperdrive.
Here's how Hyperdrive.el works under the hood.
It uses Please, the HTTP library that Adam Porter wrote, to send requests via cURL to HyperGateway.
HyperGateway is a program that Mauve Signweaver wrote, which runs a Hyperdrive node under the hood.
It also runs a local HTTP server, which accepts requests to control the node.
For example, if Hyperdrive.el wants to show a file from someone's hyperdrive, it sends the appropriate link via cURL as a GET request to HyperGateway.
HyperGateway then fetches the data from the network and returns it via cURL, via Please, back to Hyperdrive.el.
If HyperGateway already has a locally cached copy, it doesn't bother checking the network.
It just sends it straight away.
Likewise, if Hyperdrive.el wants to add a file to a hyperdrive, it sends a PUT request.
HyperGateway is not installed as part of Hyperdrive.el.
It needs to be installed as a separate, executable program.
We have plans to switch from using HyperGateway to another program that Mauve Signweaver is working on, called HyperSDK RPC.
HyperSDK RPC will give us more fine-grained control over the underlying Hyperdrive node, which will open up some new features.
For example, we'll be able to rename files atomically, diff directories between versions, and list the peers that we're currently connected to.
Here are some of the Emacs libraries that Hyperdrive.el depends on.
Hyperdrive.el uses Please to send HTTP requests to HyperGateway. Check it out.
It's on GNU Elpa.
Ewok.el is a built-in library that's documented in the Emacs Lisp manual under the rather cryptic heading Abstract Display Functions.
Really, what it does is it allows you to map a data model to some display.
What we do with it in Hyperdrive.el is, in the directory view, we map directory entry items, files and directories, to display items.
And Ewok.el makes it easy to update the display whenever the underlying data changes.
Hyperdrive.el uses persist.el to store Hyperdrive metadata about known Hyperdrives as well as version history between sessions,
so that the data is cached when you log out and log back in.
If you're going to be developing a program in Emacs for Emacs 29 or later, I recommend looking at multisession.el,
because it's built-in and it does some features that persist.el doesn't have.
Thank you for listening to this talk.
I hope you get a chance to try out Hyperdrive.el.
Here is a link to the Hyperdrive manual in the Ushen Hyperdrive.
When you go to paste this link in, you'll need to combine it back into one line.
This link is available in the Hyperdrive manual, which is available from within a browser, at this link on the Ushen website.
We also have a public conference room that you can join.
It's an XMPP multi-user chat that the Sopranica team graciously makes available for us to use.
There's also a matrix bridge, if that's more your cup of tea.
Thank you to the Emacs Conf organizers.
It's been a pleasure to participate, and I look forward to hearing the rest of the talks.
Have a good day.