All talks: https://emacsconf.org/2023/talks/
Collaborative data processing and documenting using org-babel
https://emacsconf.org/2023/talks/collab - Jonathan Hartman, Lukas C. Bossert - Track: General
Watch/participate: https://emacsconf.org/2023/watch/gen/
Q&A: after the event
IRC: https://chat.emacsconf.org/#/connect?join=emacsconf,emacsconf-gen or #emacsconf-gen on libera.chat network
Guidelines for conduct: https://emacsconf.org/conduct
See end of file for license (CC Attribution-ShareAlike 4.0 + GPLv3 or later)
----------------------------------------------------------------
Notes, discussions, links, feedback:
- I like the way you highlight the point you are talking about in real time.
- Conflict-free Replicated Data Types (CADT) :: https://github.com/emacs-straight/crdt
- !This is the future of PAD for our conference.
- Just came here to say watching two users editing the same buffer simultaneously is BLOWING MY MIND
- BLOWING MY MIND +2
- blowing my mind, too …
- WOW
- Gitlab custom-export.setup
- Truly one of the most impressive talks of the day. Congrats! Very inspiring
- Yes, indeed.
- (Lukas) Wow! Thank you. We werent sure if this is worth showing at EmacsConf because there already have been plenty of talks about literate programming and org-babel....
- Great collaborative conversation and step-wise example creates a different (and impactful) framing. Thank you!
----------------------------------------------------------------
Questions and answers go here:
- Q: How reliable it resolves the conflict? I mean, for my personal use case, for example, Sycnthing, sometimes it's not working perfectly and I had to manually edit it. How is it robust compared to syncthing?
- A (Lukas): We also faced sometimes issues that letters got mixed up. We couldnt figure out what caused it and it was not reproducable . I cannot compare it to syncthing, never used that with emacs/org-mode.
- Q: How's the security for this kind of things? I mean, if we adopt these things in our PAD, is there any, can this thing execute arbitrary (elisp) code in different people's computer? (Think like an adversary!)
- A: (Lukas) As far as we saw the code is executed on the local computer, see the part with the R-code in our video.
- (zaeph) We had plans with qhong (maintainer of crdt.el) to tunnel the connection via SSL, but we were blocked by the SSL library that shipped with Emacs, sadly. However, we did create a security policy that allowed restrictions on the execution of Elisp code. (great!)
- Q: Really nice talk and demo! You guys clearly rehearsed :). I always wonder with serial data processing sequencing like this, to what degree do the intermediate outputs need to appear inline in the text? Suppose you had 50,000 or one million rows from your initial wikidata (or similar) call. How would you handle that size of data using a collaborative, literate approach like this?
- A: (Lukas) Good question. In your local buffer there is no difference and for the collaborative partner I cannot tell. We testet it with 50 items because that was enough for demonstrating our purpose.
- noweb allows getting results of evaluation without having to put the actual data into Org buffer - just arrange the original block generating the data to have :results silent. Basically, :var foo=block-name does not require "block-name" to be evaluated in advance - it will be evaluated as necessary. AFAIU, in the talk, it is re-evaluated every time (to not have it, one would need :cache t).
- This has tremendous utility
- So it would be stored on disk and referenced by name in a subsequent block? Sounds useful.
- Not on disk - just cached within a single session. To store on disk, need to save to actual file on disk.
- Q: How do you handle the viewing of larger or really any tabular data in Emacs/Org when you want to inspect it, like the nice way tabular data is displayed inline in Rmarkdown/RStudio?
- A: (Lukas) I have no particular way of doing this.
- What about pandas data summary functionality? Can be a simple python block.
- Lukas: Jonathan is our python expert, he might answer this question.
- A: (Jonathan) If I follow, you can certainly just use DataFrame.describe() or Series.describe() to get summary statistics for a dataset - the return value would be a Series or a DataFrame, which would be displayed similiarly to how we show things here. Alternatively, DataFrame.head(n) or DataFrame.sample(n) would return a dataframe of the first n / n random lines of a dataset, and might be a way of providing the gist of a very large dataset without printing the entire table in the document.
- Would be nice to have a "summarized table" functionality in Org, that includes an abridged copy of a long table inline, but you can open it in another buffer to browse/edit the full table (ala block edit).
- Q: I'm thinking about an application for a single user, but in different platforms. In a simple case. For example, you have a buffer in your local computer, and you also want to have some files on your pad or on your phone, and you can use this CADT concept to make sure that there's not too much conflict in between different editing sections. Do you think this is a good idea? I mean, compared to purely relying on Syncthing, which sometimes I feel is unreliable for resolving those conflicts.
- A: (Lukas) This sounds very interesting and could beneficial for contiously working on things.
- Q:
----------------------------------------------------------------
Next talks:
Questions/comments related to EmacsConf 2023 as a whole? https://pad.emacsconf.org/2023
----------------------------------------------------------------
This pad will be archived at https://emacsconf.org/2023/talks/collab after the conference.
Except where otherwise noted, the material on the EmacsConf pad are dual-licensed under the terms of the Creative Commons Attribution-ShareAlike 4.0 International Public License; and the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) an later version. Copies of these two licenses are included in the EmacsConf wiki repository, in the COPYING.GPL and COPYING.CC-BY-SA files (https://emacsconf.org/COPYING/)
By contributing to this pad, you agree to make your contributions available under the above licenses. You are also promising that you are the author of your changes, or that you copied them from a work in the public domain or a work released under a free license that is compatible with the above two licenses. DO NOT SUBMIT COPYRIGHTED WORK WITHOUT PERMISSION.